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Forest  Service  decision-making  processes  frequently 
need  information  that  cannot  be  adequately  met  witii 
existing  knowledge  and  data,  yet  field  data  collection 
is  often  infeasible  or  untimely.  Certain  nonfield 
methods  can  provide  defensible,  timely  information. 
Delphi  is  one  such  method. 

Eleven  elk  habitat  experts  served  as  participants  in 
our  Delphi  study.  Each  participant  evaluated  the  sum- 
mer cover  quality  of  171  elk  habitat  settings  by  rating 
them  on  a  scale  of  1  to  7  and  recording  their  reasons 
for  the  ratings.  These  evaluation  bases  were  then 
made  known  to  all  participants,  who  could  then  reas- 
sess their  original  judgments.  This  process  of  evalua- 
tion, feedback,  and  reevaluation  was  made  (iterated) 
three  times  to  obtain  final  summer  cover  ratings,  and 
again  to  obtain  summer  forage  quality  ratings.  Study 
results  consist  of  habitat  quality  ratings  and  evalua- 
tions of  the  Delphi  process. 

Habitat  quality  ratings  ranged  from  "very  low"  to 
"very  high,"  with  frequency  forming  a  bell  shape  for 
both  summer  cover  and  forage  quality.  The  quality  of 
elk  summer  cover  was  highly  related  to  the  structure 
of  site  vegetation— tree  size  and  stand  density.  Sum- 
mer forage  quality  was  highly  related  to  the  type  of 
vegetation  on  the  site— vegetation  cover  type  and  for- 
est habitat  type.  Discriminant  functions  were  devel- 
oped capable  of  correctly  classifying  up  to  90  percent 
of  the  sites  into  habitat  quality  classes  based  on  site 
characteristics. 

We  also  evaluated  the  Delphi  process.  The  11 
research  participants  each  devoted  an  average  of 
slightly  over  13  hours  to  this  study.  Significant 
improvements  in  participant  agreement  (consensus)  on 
habitat  quality  ratings  were  achieved  by  means  of  the 
three-iteration  process,  but  the  median  rating  for  each 
of  the  171  settings  rarely  changed.  No  statistically  sig- 
nificant difference  was  found  in  consensus  between 
participants  on  the  basis  of  their  occupational  status, 
expertise,  or  experience.  However,  a  small  but  statisti- 
cally significant  difference  was  found  between  habitat 
quality  ratings  of  managers/administrators  and 
researchers/academics.  Participants  indicated  satisfac- 
tion with  the  Delphi  process  and  confidence  in  the 
habitat  quality  ratings. 
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The  Delphi  Method:  Application 
to  Elk  Habitat  Quality 


Ervin  G.  Schuster 
Sidney  S.  Frissell 
Eldon  E.  Baker 
Robert  S.  Loveless,  Jr. 


INTRODUCTION 

Managers  in  the  USDA  Forest  Service,  like  those  in 
other  pubHc  resource  management  organizations,  are 
required  to  base  decisions  on  careful  evaluations  of 
management  actions.  To  ensure  thoroughness  in  evalua- 
tion, a  wide  range  of  consequences  are  typically 
assessed,  such  as  national  economic  development, 
environmental  quality,  regional  development,  and  social 
well-being.  To  ensure  consistency,  comparability,  and 
integrity  in  evaluations,  formal  procedures  are  adopted, 
sometimes  requiring  the  use  of  sophisticated  mathemati- 
cal models.  As  a  result,  more,  better,  and  different  kinds 
of  data,  relationships,  facts,  and  figures  are  needed. 
More  than  ever  before,  information  fuels  the  furnaces  of 
decision  making. 

But  like  money,  information  does  not  grow  on  trees. 
Information  must  be  gathered  by  people  using  some 
process.  Resource  scientists  extract  information  from 
data  gathered  through  use  of  the  scientific  method, 
along  with  rigorously  controlled  experimental  designs 
and  meticulously  collected  measurements.  Unfortunately, 
the  constraints  of  scientific  research  often  lead  to  infor- 
mation that  is  "too  little,  too  late."  When  review  of  the 
scientific  literature  fails  to  provide  needed  information, 
resource  managers  gather  their  own,  frequently  employ- 
ing procedures  that  emphasize  speed  and  simplicity— 
"quick  and  dirty."  While  information  so  generated  satis- 
fies the  data  requirements  of  mathematical  models,  too 
often  it  lacks  the  scientific  credibility  needed  to  defend 
results. 

Choice  between  information-generating  methods  need 
not  be  lirnited  to  "too  little,  too  late"  or  "quick  and 
dirty."  A  number  of  techniques  currently  exist  whereby 
timely  and  credible  information  can  be  developed, 
namely  the  Nominal  Group  Technique  and  the  Delphi 
Method.  Both  techniques  produce  information  by  syste- 
matically exploiting  the  opinions,  ideas,  experience,  and 
knowledge  of  individuals,  and  neither  technique  involves 
onsite  field  measurements.  The  Delphi  Method  differs 
from  the  Nominal  Group  Technique  by  not  requiring  a 
face-to-face  meeting  of  the  group  participants.  This  fea- 
ture has  two  important  implications.  First,  participants 
are  not  unduly  influenced  by  the  status  or  personal  style 
of  any  other  participant.  Second,  because  participants 
need  not  congregate  in  one  place,  Delphi  avoids  the  time 


and  expense  of  travel  and,  consequently,  can  be  rela- 
tively inexpensive  to  apply. 

Delphi  is  not  a  panacea  for  all  information  woes. 
Indeed,  there  are  instances  where  Delphi  is  inappropriate 
or  where  some  other  technique  may  be  preferable.  But 
Delphi  does  hold  the  promise  of  being  able  to  provide 
quality  information  over  a  wide  range  of  topics  impor- 
tant to  natural  resource  managers.  The  biggest  impedi- 
ment to  application  of  Delphi  may  well  be  its  lack  of 
exposure.  Resource  managers  are  simply  unacquainted 
with  it. 

This  paper  reports  on  the  results  of  a  study  designed 
both  to  gather  important  information  through  Delphi 
and  to  evaluate  several  key  features  of  the  Delphi  proc- 
ess. We  intend  to  present  information  developed,  and 
also  describe  and  discuss  Delphi  so  as  to  make  resource 
managers  more  comfortable  with  it,  aware  of  its  poten- 
tial, and  willing  to  consider  it  along  with  other  informa- 
tion gathering  tools. 

This  study  focuses  on  the  quality  of  elk  habitat  in 
western  Montana  and  the  Delphi  Method.  We  not  only 
wanted  to  better  understand  elk  habitat  but  the  Delphi 
Method  as  well.  Elk  habitat  quality  data  were  developed 
through  a  Delphi  process  and  then  these  data  were 
assessed:  (1)  to  determine  the  effect  of  vegetation  type 
and  structure  on  habitat  quality;  (2)  to  determine  the 
effect  of  site-specific  factors,  such  as  temperature  and 
moisture,  on  habitat  quality;  and  (3)  to  develop  a  tech- 
nique for  assigning  a  quality  rating  to  a  habitat  setting. 
These  data  were  also  assessed  from  several  standpoints 
pertaining  to  Delphi— effect  of  iteration  and  number  of 
participants  on  consensus,  participant  selection,  and 
more. 

METHODS 

Because  the  methods  used  in  this  study  are  more  con- 
fusing than  complicated,  it  is  best  to  separate  discussion 
of  the  Delphi-oriented  methods  and  the  analytical 
methods  used.  The  discussion  is  somewhat  artificial  in 
that  while  these  methods  are  presented  separately,  they 
are  in  reality  interconnected. 

Delphi  Methods 

Delphi  was  developed  by  the  RAND  Corporation  as  a 
systematic  tool  for  harnessing  group  knowledge  pertain- 
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ing  to  a  wide  range  of  program  planning  matters 
(Dalkey  1969).  Delphi  has  three  main  features  (Delbecq 
and  others  1975):  it  uses  written  responses,  participants 
typically  remain  anonymous,  and  the  process  uses  a 
series  of  questionnaires  to  clarify  areas  of  agreement  and 
disagreement.  Delphi  aims  to  promote  group  consensus 
on  some  topic  or  issue  by  extending  the  adage  that  "two 
heads  are  better  than  one."  Several  heads  are  used.  Con- 
sensus is  not  achieved  immediately,  if  ever.  Improve- 
ments in  consensus  result  from  an  iterative  process 
wherein  participants  assess  an  issue,  document  or  justify 
their  assessment,  and  are  given  opportunity  to  reassess 
their  earlier  position  in  light  of  information  from  other 
participants. 

The  several  methodological  components  involved  in 
this  study  are  illustrated  in  figure  1.  This  and  the  next 
section  discuss  each  component:  the  topic  and  why  it 
was  selected;  how  participants  were  identified  and 
selected:  the  questionnaire  or  data-gathering  instrument; 
how  feedback  was  provided  to  the  participants:  the  exit 
interview-  questionnaire  used  to  conclude  data  collection: 
and  a  general  description  of  the  methods  used  to  analyze 
data. 

The  Topic— Elk  Habitat.— The  range  of  topics  to  which 
Delphi  can  be  applied  is  wide.  Although  almost  any 
topic  would  do.  we  wanted  to  assess  one  representing  an 
actual  information  need  relevant  to  forest  managers.  We 
also  wanted  a  topic  of  sufficient  scope  and  complexity  to 
be  judged  nontrivial,  yet  manageable.  We  considered 
topics  from  timber  management,  recreation  management, 


wildlife  management,  and  so  on.  The  one  we  eventually 
selected  involved  measuring  the  quality  of  elk  {Cervus 
elaphus  nelsoni]  summer  forage  and  summer  cover  in 
Montana  west  of  the  Continental  Divide.  Although  win- 
ter range  is  often  considered  a  limiting  factor  for  elk. 
summer  habitat  was  selected  for  stud}-  because  National 
Forest  System  lands  contain  a  substantial  portion  of  elk 
range  that  is  predominantly  summer  range.  Thomas 
(1979)  describes  forage  as  vegetation  used  for  food  by 
wildlife:  cover  is  vegetation  used  by  wUdlife  for  protec- 
tion from  predators,  to  ameliorate  conditions  of  weather, 
or  in  which  to  reproduce.  No  distinction  was  made 
between  thermal  (from  heat)  and  hiding  Ifrom  predators) 
cover. 

Montana  west  of  the  Continental  Divide  was  selected 
as  the  study's  geographical  scope  because  it  is  a  major 
elk-producing  area,  and  elk  habitat  considerations  are 
often  critical  in  forest  management.  The  area  is  also 
large  enough— western  Montana  is  roughly  the  size  of 
West  Virginia— that  obtaining  similar  information  from 
field  measurements  would  be  essentially  infeasible. 
Finally,  the  study  area  is  relatively  homogeneous 
ecologically. 

Participant  Selection.— As  with  any  group  process, 
Delphi  requires  participation  by  a  set  of  individuals.  The 

type  of  participants  used  in  a  Delphi  project  should 
depend  on  the  nature  of  the  topic  being  addressed.  The 
topic  of  elk  habitat  quality  called  for  participants  with  a 
specialized,  technical  knowledge  consistent  with  the  term 
"experts,"  recognized  authorities  on  elk  habitat.  Unfor- 
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Figure  1.— Flowchart  of  study  methodological  components  and  their  linkages. 
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tunately,  because  there  is  no  directory  of  elk  habitat 
experts,  we  needed  to  identify  them  ourselves. 

Numerous  direct  and  indirect  ways  exist  for  identify- 
ing elk  habitat  experts.  We  developed  a  composite  pi'oce- 
dure  that  actually  consisted  of  several  methods.  The 
approach  used  could  be  called  "networking,"  being  simi- 
lar to  that  used  in  sociological  studies  to  identify  a  com- 
munity's power  structure  (see,  for  example,  Domhoff 
1978). 

Based  on  the  study  team's  perception  of  people  with 
substantial  knowledge  in  elk  habitat,  a  "starter"  was 
selected.  We  attempted  to  identify  a  knowledgeable  per- 
son who  would  not  likely  be  judged  by  others  as  an 
expert.  That  person  was  interviewed  and  asked  to  list  all 
experts  on  elk  habitat  management  in  western  Montana: 

.  .  .  Please  identify  all  individuals  that  you  consider 
to  be  experts.  They  may  be  from  any  occupational 
status,  such  as  managers,  teachers,  administrators, 
researchers,  or  from  the  lay  public.  They  need  not 
be  physically  located  in  western  Montana,  but  must 
be  knowledgeable  about  it.  Please  include  yourself, 
if  appropriate. 

The  starter  was  then  instructed  to  rank  the  identified 
persons  in  terms  of  the  degree  of  expertise,  from  most  to 
least,  paying  particular  attention  to  correctly  specifying 
the  order  of  the  top-three  persons.  Each  of  the  top-three 
experts  from  the  starter's  list  was  then  contacted  and 
asked  to  identify  and  rank  all  experts,  again  paying  par- 
ticular attention  to  the  order  of  the  top  three.  All  new 
top-three  experts  were  then  contacted  and  the  process 
was  continued  until  no  new  top-three  experts  were  iden- 
tified. This  procedure  was  adopted  so  that  we  could  later 
compare  the  ability  of  different  approaches  to  identify 
experts. 

To  ensure  procedural  consistency,  a  standard  interview 
format  was  always  used.  All  interviews  were  either  con- 
ducted in  person  or  by  telephone.  Each  individual  con- 
tacted was  free  to  use  personal  criteria  for  judging  an 
expert.  As  was  the  case  throughout  the  entire  study, 
individuals  were  given  the  assurance  of  anonymity,  that 
their  identity  and  organizational  affiliation  would  be  held 
in  the  strictest  confidence. 

The  Data-Gathering  Instrument.— Any  Delphi  exercise 
must  use  some  method  to  address  the  topic  and  obtain 
measurements— a  data-gathering  instrument.  Although 
in  Delphi  this  instrument  is  called  a  questionnaire,  it 
need  not  look  like  a  normal  questionnaire.  In  our  case, 
the  data-gathering  instrument  consisted  of  pictorial  site 
representations  (PSR's),  a  scaling  or  classification  proce- 
dure, and  a  set  of  instructions. 

If  this  were  a  field  study,  actual  measurements  would 
be  made  on  a  site.  The  PSR's  depicted  conditions  that 
might  be  found  on  an  actual  site.  We  identified  171 
specific  elk  habitat  settings  (discussed  later)  representing 
various  combinations  of  (1)  vegetation  cover  type,  and  if 
forested,  (2)  groupings  of  forest  habitat  types  (Pfister 
and  others  1977),  (3)  tree  size,  and  (4)  stand  density  or 
stocking.  For  example,  one  of  the  171  settings  was  a 
well-stocked  stand  of  sawtimber-sized  lodgepole  pine 
trees  growing  on  a  spruce  habitat  type.  There  were  171 
PSR's,  one  for  each  elk  habitat  setting. 


Figure  2  shows  that  each  PSR  consisted  of  a  written 
description  of  the  vegetation  and  its  environment,  a 
black  and  white  sketch  showing  a  cross-sectional  view  of 
the  vegetation,  and  an  illustrative,  color  photocopy.  Sub- 
stantial thought  went  into  the  content  and  format  of  the 
PSR.  The  result  represented  a  compromise  between  what 
was  desired  and  what  was  physically  and  financially  pos- 
sible. For  example,  we  wanted  to  use  color  prints  exclu- 
sively, but  they  were  prohibitively  expensive.  Similarly, 
while  we  wanted  to  use  two  photographs  (near  and  far 
view),  we  could  not  accumulate  a  satisfactory  set  of  pho- 
tographs for  all  171  elk  habitat  settings.  Therefore,  only 
near-view  photographs  were  used. 

The  study  team  produced  the  PSR's.  We  obtained 
many  color  photographs  from  existing  print  and  slide 
collections.  If  the  needed  photographs  were  either  inade- 
quate or  not  available,  a  study  team  member  traveled  to 
appropriate  sites  and  took  the  photographs.  An  artist 
knowledgeable  in  vegetation  structure  drew  the  black 
and  white  sketches.  The  written  descriptions  were  based 
on  Pfister  and  others  (1977).  Initial  descriptions  were 
reviewed  and  revised  by  Robert  D.  Pfister.  We  assem- 
bled all  materials  into  a  master  set.  A  local  commercial 
operator  duplicated  the  master  set  on  heavyweight  bond 
paper  for  each  participant. 

A  scaling  procedure  was  designed  to  classify  PSR's 
into  habitat  quality  groups,  thereby  providing  a  meas- 
urement of  the  quality  of  each  PSR  in  terms  of  elk  sum- 
mer cover  and  forage.  We  chose  to  use  a  modified  Q-sort 
Method  (SeUtiz  and  others  1959)  and  a  seven-level  scale. 
We  sent  each  Delphi  participant  a  set  of  171  randomly 
ordered  PSR's.  The  participants  were  told  to  look 
through  the  PSR's  to  get  a  feel  for  the  range  of  condi- 
tions with  which  they  were  to  work.  They  were  then 
instructed  to  place  each  PSR  into  one  of  thi-ee  groups 
according  to  elk  habitat  quality— high,  medium,  and 
low— initially  for  cover  and  then  later  in  the  study  for 
forage.  The  high  and  low  quality  groups  were  then  fur- 
ther subdivided  into  two  groups,  and  the  middle  group 
was  subdivided  into  three  groups.  Thus,  each  PSR  was 
systematically  placed  into  one  of  seven  groups.  Numeri- 
cal codes  were  assigned  to  the  groups  of  PSR's  accord- 
ing to  the  following  definitions: 


Very  low 

(1) 

—  The  worst  sites,  with  virtually 

no  habitat  value 

Low 

(2) 

—  Poor,  but  with  minimal  elk 

habitat  value 

Low  medium 

(3) 

—  The  poorest  of  the  medium  sites 

Medium 

(4) 

—  Adequate  or  average 

High  medium 

(5) 

—  The  best  of  the  medium  site 

High 

(6) 

—  Somewhat  less  than  best  but 

still  high  quality 

Very  high 

(7) 

—  The  best  sites 

Participants  were  invited  to  adjust  PSR's  among  groups 
to  ensure  that  each  PSR  ultimately  corresponded  to  the 
group's  definition. 

Participants  were  provided  with  other  instructions 
that  promoted  consistency.  The  following  instructions 
are  typical,  though  specifically  pertaining  to  summer 
cover  measurements: 
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1.  Your  ratings  are  for  elk  habitat  only. 

2.  Our  concern  at  this  point  is  for  elk  summer  cover 
quality  in  western  Montana.  Do  not  complicate  the 
situation  by  attempting  to  visualize  winter  cover, 
summer  food  value,  and  so  forth.  We  are  only  in- 
terested in  how  valuable  each  site  is  for  providing 
cover  for  elk  during  midsummer,  prior  to  any  range 
shifts  that  might  occur  due  to  the  onset  of  the  rut. 

3.  Remember  that  each  PSR  illustrates  a  specific  site 
condition.  Try  to  keep  the  general  site  characteris- 
tics in  mind  when  doing  the  rating  and  don't  get 
led  astray  by  some  minor  peculiarity  of  a  particular 
photo. 

4.  We  are  assuming  there  are  no  active  roads  in  the 
vicinity  and  there  has  been  no  timber  harvesting  or 
other  stand  manipulation. 

5.  The  PSR's  are  designed  specifically  to  represent  a 
range  of  conditions  common  in  the  forests  of  Mon- 
tana west  of  the  Continental  Divide  and  should  be 
evaluated  in  terms  of  that  area. 

Each  participant  was  also  provided  a  form  on  which  to 
keep  track  of  the  time  spent  on  aspects  of  the  study. 
Participants  were  instructed  to  complete  sill  tasks  and 
return  all  material  (in  a  stamped  return  envelope)  by  a 
specified  date,  usually  about  2  weeks.  Finally,  par- 
ticipants were  instructed  not  to  discuss  this  study  or 
their  involvement  with  others. 

Feedback.— A  Delphi  process  is  designed  to  promote 
group  agreement  (consensus)  on  a  topic.  This  is  accom- 
plished by  "controlled  feedback"  and  "iterations."  The 
feedback  process  allows  participants  to  share  with  each 
other  their  personal  basis  for  the  judgments.  The  itera- 
tion (repeating)  process  allows  participants  to  reassess 
their  prior  individual  judgments  in  light  of  the  summa- 
rized information  (reasons,  opinions)  provided  by  all  par- 
ticipants. There  are  two  important  aspects  to  the  feed- 
back process.  One  relates  to  acquisition  of  information 
by  the  study  team  and  the  other  deals  with  presentation 
of  summarized  information  to  the  participants.  There  is 
no  standard  format  to  provide  feedback  in  Delphi.  It 
must  be  tailored  to  the  situation. 

The  study  team  devoted  many  hours  to  discussing  al- 
ternative types  of  feedback.  We  selected  two.  One  type 
involved  summarizing  the  bases  participants  used  for 
their  judgment.  The  other  type  of  feedback  involved 
providing  each  participant  with  some  quantitative  infor- 
mation on  how  the  PSR's  were  rated  by  other  par- 
ticipants. Because  all  ratings  were  stored  in  computer 
data  files,  the  study  team  already  had  information  on 
how  the  PSR's  were  rated.  The  reasons  given  by  par- 
ticipants for  rating  PSR's  were  obtained  by  a  short 
questionnaire.  When  completed,  the  questionnaire 
provided  the  study  team  with  a  listing  of  the  common 
characteristics  associated  with  the  PSR's  in  each  cate- 
gory. The  following  directions  were  provided: 

...  In  the  spaces  provided,  please  list  (complete 
sentences)  up  to  three  statements  you  would  use 
to  characterize  the  common  elements  represented 
by  PSR's  in  quality  categorv  1,  quality  category 
2,  and  so  forth.  For  example,  "all  PSR's  in  cate- 


gory 1  are  very  open,  young  stands  with  little 
crown  cover." 

Collecting  feedback  information  was  considerably  eas- 
ier than  communicating  it  back  to  the  participants. 
After  all  sets  of  reasons  (common  elements)  were 
received  from  each  participant,  the  reasons  were  evalu- 
ated by  "content  analysis"  procedures  (see  Bowers  1970) 
to  further  identify  common  features.  After  several  trials, 
content  analyses  were  judged  too  unwieldy  and  were  dis- 
continued. To  condense  exhaustive  amounts  of  written 
information  into  a  manageable  form,  the  study  team 
chose  to  summarize  stated  reasons  in  the  form  of  sum- 
mary evaluations.  One  set  of  reasons  was  developed  and 
provided  to  each  participant.  Each  set  consisted  of  seven 
pages,  one  for  each  rating  category.  Each  page  contained 
information  in  five  categories— tree  cover,  tree  size,  habi- 
tat type,  cover  type,  and  other  features.  The  order  in 
which  these  categories  appeared  on  a  summary  evalua- 
tion page  was  varied  so  as  to  preclude  any  bias  in- 
troduced by  a  consistent  order.  A  partial  illustration  of  a 
page  follows: 

Category  1  -  Very  Low  -  Forage 
I.    Tree  cover:* 

Nonstocked-  20% 
Poorly  stocked-  35% 
Medium  stocked-  10% 
Well  stocked-  0% 


V.  Other: 

Poor  soils-  20% 
Vegetation  dries  early - 


20% 


*Percentage  indicates  proportion  of  experts  using  or 
mentioning  the  cliaracteristic  in  their  reasons. 

The  study  team  carefully  considered  how  to  best  com- 
municate quantitative  information  about  how  the  PSR's 
were  rated.  The  issue  was  not  to  provide  a  comparison 
showing  PSR  ratings  and  how  an  individual  participant 
related  to  others.  Rather,  the  issue  pertained  to  the 
amount  of  detail.  Arguments  against  detail  were  based 
on  the  fear  of  exerting  too  much  pressure  on  the  par- 
ticipants to  conform.  Arguments  in  favor  of  detail  were 
based  on  the  premise  that  a  participant  is  more  likely  to 
modify  a  judgment  if  that  judgment  is  perceived  to  be 
nonconforming.  We  adopted  the  more  detailed  approach. 
Each  participant  was  provided  a  computer  printout  of 
the  ratings  (rating  printout),  as  partially  illustrated: 

All  experts  combined  Your 

PSR      Low      High      Median  rating  Difference 

1  2  4  4  3  -1 

2  4  4  4  4  0 


171        6  7  6  6  0 

The  summary  evaluations  and  the  rating  printouts 
constituted  the  feedback  provided  to  each  participant 
after  the  initial  rating.  Consistent  with  contemporary 
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Delphi  literature  and  application,  each  interaction  (in- 
cluding the  first)  with  participants  constitutes  an  "itera- 
tion." By  convention,  the  first  iteration  has  no  feedback. 

In  principle,  the  number  of  iterations  used  in  a  Delphi 
project  is  open-ended,  determined  on  the  basis  of  some 
prespecified  level  of  consensus  desired.  The  iterative 
process  continues  until  the  consensus  standard  is  met. 
But  because  we  wanted  to  measure  the  effect  that  the 
number  of  iterations  had  on  consensus,  several  iterations 
were  needed.  Because  we  did  not  believe  the  experts 
would  agree  to  participate  in  a  process  with  no  finite 
limits,  we  decided  to  use  three  iterations  for  both  the  elk 
summer  cover  phase  and  the  forage  phase.  This  amount 
represented  our  judgment  as  to  the  minimum  number  of 
iterations  needed  to  measure  the  effect  of  iterations  on 
final  consensus  and  the  maximum  burden  we  could 
reasonably  place  on  the  participants. 

All  feedback  materials,  and  the  171  PSR's  as  previ- 
ously sorted,  were  returned  to  each  participant  to  begin 
the  second  iteration.  The  second  iteration  consisted  of 
each  expert  reassessing  first  iteration  judgments  in  light 
of  the  feedback  materials.  Individual  PSR's  were  shifted 
from  one  habitat  quality  group  to  another.  When  reas- 
sessments were  completed,  participants  again  recorded 
their  reasons  and  returned  the  materials.  The  process 
was  iterated  for  a  third  time,  thus  completing  the  sum- 
mer cover  phase.  The  entire  three-iteration  process  was 
repeated  for  summer  forage. 

Exit  Interview  Questionnaire.— After  all  habitat  qual- 
ity data  had  been  gathered,  the  study  team  wanted  to 
know  the  participants'  reactions  to  the  Delphi  process. 
What  were  the  strengths  and  weaknesses?  Were  seven 
Q-sort  groups  too  many?  Was  the  feedback  adequate? 
These  and  other  concerns  were  addressed  in  an  exit 
interview  questionnaire,  available  on  request  from  the 
authors.  The  questionnaire  was  mailed  to  participants. 
Upon  return  of  all  questionnaires,  data  collection  was 
complete  and  final  analyses  were  started. 

Analytical  Methods 

This  section  briefly  describes  the  variables,  methods, 
and  conventions  used  to  design  the  study  and  analyze 
the  data.  It  does  not  provide  enough  detail  to  duplicate 
the  analyses,  but  rather  enough  for  the  reader  to  develop 
a  general  comprehension.  More  details  pertaining  to  ana- 
lytical methods  will  be  provided  later,  as  appropriate, 
along  with  study  results. 

Variables.— This  study  used  two  classes  of  variables- 
primary  and  secondary.  The  primary  variables  resulted 
directly  from  the  main  thrust  of  the  study,  were  the 
basis  for  the  study's  experimental  design,  and  specified 
the  way  PSR's  were  characterized.  Table  1  identifies  and 
defines  primary  dependent  and  independent  variables. 
All  variables  are  measured  on  either  nominal  or  ordinal 
scales,  with  analytical  impUcations  that  will  be  discussed 
later. 

The  primary  dependent  variable  used  in  this  study 
consists  of  elk  summer  cover  and  forage  quality  ratings. 
For  purpose  of  conducting  statistical  or  numerical  ana- 
lyses, the  habitat  quality  ratings  were  assigned  numeri- 
cal values  ranging  from  1  for  very  low  to  7  for  very 


high.  Measures  of  central  tendency  for  this  type  of  ordi- 
nal scale  data  include  the  median  (middle)  and  mode 
(most  frequent)  rating,  but  exclude  the  arithmetic  mean. 
This  study  used  the  median  rating  to  reflect  a  PSR's 
final,  overall  rating.  The  median  was  also  used  as  a  basis 
to  reflect  both  variation  in  habitat  ratings  and  con- 
sensus. We  used  the  average  or  mean  deviation  from  the 
median  (MDM)  as  our  measure  of  variation  and  con- 
sensus. This  is  calculated  by  summing  the  (absolute) 
deviations  from  the  median  over  all  participants  and 
dividing  by  the  number  of  participants.  The  higher  this 
number,  the  more  variation  around  the  median.  MDM  is 
comparable  to  an  unsquared  version  of  the  "variance" 
used  in  conjunction  with  a  mean.  If  all  11  experts  rated 
a  situation  as  "medium"  (4),  the  median  would  be  4  and 
the  mean  deviation  from  the  median  would  be  0,  thus 
achieving  unanimity,  complete  agreement,  or  perfect 
consensus. 

The  forest  habitat  type  group  variable  was  created  by 
the  study  team.  To  use  all  of  the  individual  forest  habi- 
tat types  would  have  been  too  cumbersome  and  unneces- 
sarily specific  for  our  purpose.  Therefore,  the  individual 
forest  habitat  types  identified  by  Pfister  and  others 
(1977)  were  grouped  into  18  habitat  type  groups.  These 
are  presented  in  appendix  A,  along  with  their  constitu- 
ent forest  habitat  types.  The  grouping  represents  a 
slight  modification  and  updating  of  an  elk  habitat  group 
scheme  developed  by  Forest  Service  wildlife  managers 
for  use  in  western  Montana  (USDA  Forest  Service  1980). 

While  the  forest  habitat  type  group  variable  and  the 
vegetation  cover  type  variables  can  be  used  separately, 
we  frequently  combined  them.  These  combinations  cor- 
respond to  major  natural  occurrences  of  habitat  type 
groups  and  cover  types  in  western  Montana.  Table  2 
shows  the  37  naturally  occurring  combinations  of  our  18 
forest  habitat  type  groups  and  12  vegetation  cover 
types.  Depending  on  the  purpose,  some  analyses  used 
the  combinations,  while  others  used  either  the  18  habitat 
type  groups  or  12  cover  types  separately. 

Similarly,  combinations  of  tree  size  class  and  stand 
density  classes  were  developed.  Tree  size  (TS)  was 
divided  into  three  classes:  seedling/ sapling  (S/S),  poletim- 
ber  (Pole),  and  sawtimber  (Saw).  Stand  density  (SD)  was 
also  divided  into  three  classes:  poor,  medium,  and  weU 
stocked.  The  ideal  would  have  been  an  elk  habitat 
represented  by  all  possible  combinations  of  tree  size  and 
stand  density  classes.  This  proved  infeasible  for  two  rea- 
sons. First,  not  all  combinations  of  size  and  density 
occur  naturally.  An  example  is  the  poorly  stocked 
seedling-sapling  stands  of  western  redcedar,  a  combina- 
tion that  basically  does  not  naturally  exist.  Second,  we 
wanted  to  keep  the  participants'  workload  within  reason, 
and  all  combinations  seemed  unreasonable.  To  simplify 
matters,  we  adopted  five  combinations,  instead  of  the 
potential  nine: 

Stand  Tree  size  class 

density  class     S/S     Pole  Saw 
Poor  X  X 

Medium  X 
WeU  X  X 
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Table  1.— Description  of  dependent  and  independent  variables  used  in  study 


Type  of 
variable 


Dependent 


Independent 


Independent 


Independent 


Independent 


Name  of 
variable 


Cover  quality 
and 

forage  quality 


Forest  hiabitat 
type  groups 

Vegetation 
cover  types 

Tree  size 
class 


Stand 

density 

class 


Measure- 
ment 
scale 


Ordinal 


Nominal 


Nominal 


Ordinal 
or 

nominal 


Ordinal 
or 

nominal 


Code  or 
attribute 


Definitions 


1  or  VL  Very  lov^^:  v^/orst  sites,  with  virtually  no  hiabitat  value 

2  or  L  Low.  poor,  but  with  minimal  elk  habitat  value 

3  or  LM  Low  medium:  poorest  of  the  medium  sites 

4  or  M  Medium:  adequate  or  average 

5  or  HM  High  medium:  best  of  the  medium  sites 

6  or  H  High:  somewhat  less  than  the  best  but  still  high  quality 

7  or  VH  Very  high:  the  best  sites 

1  -18  Groups  of  forest  habitat  types.  (An  aggregation  of  all 

land  areas  potentially  capable  of  producing  similar 
plant  communities  at  climax;  Pfister  and  others  1977) 

1  -12  Existing  overstory  vegetation.  If  forested,  based  on 

species  forming  a  plurality  of  live-tree  stocking 
(Green  and  Setzer  1974) 

—  A  classification  of  forest  land  based  on  size  classes 
of  growing  stock  trees  on  the  area 

—  Stands  at  least  16.7  percent  stocked  with  growing  stock 
trees,  with  half  or  more  of  stocking  .  .  . 

Saw-  ...  in  sawtimber  or  poletimber  trees,  and  sawtimber 

timber  stocking  at  least  equal  to  poletimber  stocking  (saw- 

timber trees  are  >9.0  inches  dbh) 

Pole-  ...  in  poletimber  and/or  sawtimber  trees,  and  with 

timber  poletimber  stocking  exceeding  that  of  sawtimber  (pole- 

timber  trees  are  >5.0  to  <9.0  inches  dbh) 

Seedling         ...  in  saplings  or  seedlings  (seedling  and  sapling 

and  trees  are  <5.0  inches  dbh) 

sapling 

—  A  classification  of  forest  land  based  on  the  density  of 
trees  on  the  area 

Poor  11  to  40  percent  crown  cover,  or  16.7  to  60.0  percent 

normal  stocking 

Medium  41  to  70  percent  crown  cover,  or  60  to  100  percent 

normal  stocking 

Well  71  to  100  percent  crown  cover,  or  100  to  133  percent 

normal  stocking 


Because  all  treatment  combinations  are  not  present,  this 
study  used  an  incomplete  block  design.  Depending  on 
the  purpose,  some  analyses  used  the  combinations  of 
size  and  density  classes,  while  others  used  size  or  den- 
sity classes  individually,  or  both. 

Let  us  recap  the  origins  of  the  171  PSR's.  Of  the  37 
combinations  of  habitat  type  groups  and  cover  types, 
three  were  nonforested,  and  the  notions  of  tree  size  and 
stand  density  were  not  applicable.  These  nonforested 
combinations  account  for  three  PSR's.  Of  the  remaining 
34  combinations,  one  occurs  only  in  medium  and  well- 
stocked  stands;  this  situation  resulted  in  three  PSR's. 
The  remaining  33  combinations  could  be  represented  by 
all  five  size  and  density  combinations.  These  account  for 
165  (5  X  33)  PSR's.  In  total,  we  used  171  (3  +  3  +  165) 
PSR's. 

A  large  number  of  secondary  variables  were  also  used 
in  supporting  analyses: 


Secondary  Secondary 
dependent  variables  independent  variables 

Mean  deviation  from  median  Iteration 
Exit  interview  results  Number  of  participants 

Time  record  results  Participant  characteristics 

Habitat  type  components 

These  variables  were  used  both  in  conjunction  with  pri- 
maiy  valuables  and  separate  from  primaiy  variables.  Many 
variables  Listed  are  actually  groups  of  more  specific  varia- 
bles. For  example,  the  answer  to  each  question  on  the  exit 
interview  questionnaire  represented  a  potential  dependent 
variable.  Similarly,  for  several  analyses  we  used  six  compo- 
nents of  habitat  type  as  candidate  independent  variables- 
such  as  elevation,  moisture,  and  temperature.  Some  secon- 
dary variables  were  measured  on  a  nominal  scale,  such  as 
habitat  type  components.  But  others  were  measured  on  a 
ratio  scale,  such  as  time. 
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Ordinal-Level  Data  Interval-Level  Analyses.— Table  1 
indicates  that  habitat  quality  is  measured  on  a  seven-level 
ordinal  scale— very  low  to  very  high.  Considerable  con- 
troversy exists  concerning  the  appropriate  statistical  tests, 
models,  and  arithmetic  operations  that  can  be  performed 
on  ordinal  data  (see  Kirk  1972).  Specifically,  researchers 
who  have  ordinal-scale  data  would  like  to  apply  analytical 
techniques  compatible  with  interval-scale  or  ratio-scale 
data.  Those  techniques  offer  more  choices  and  can  extract 
more  data.  Without  belaboring  this  controversy,  the  oppos- 
ing viewpoints  are:  (1)  under  no  circumstances  can  interval- 
scale  or  ratio-scale  operations  be  applied  to  ordinal-scale 
data,  and  (2)  go  ahead,  do  it.  The  study  team  struck  a 
compromise.  Interval-level  statistics  were  used  for  testing 
statistical  hypotheses.  But  ordinal-level  statistics  were 
used  for  describing  data  and  study  results.  Consequently, 
we  used  medians  (appropriate  for  ordinal-level  data),  not 
means,  to  describe  habitat  quality  results.  But  we  also 
used  regression  and  other  analyses  (appropriate  for 
interval-level  data)  to  investigate  underlying  habitat  qual- 
ity relationships. 

Analyses  Used.— Data  files  were  maintained  and  ana- 
lyzed through  the  services  of  the  University  of  Montana 
Computer  Center  and  its  DEC-20  computer.  The  main  ana- 
lyses performed  were  analysis  of  variance,  linear  regression 
analysis, " and  discriminant  analysis.  Statistical  analyses 
and  data  manipulation  used  computer  software  available 
through  SPSS-X  and  Minitab.  Except  for  the  method  used 
for  interpolation,  all  other  analyses  were  conducted  in  con- 
ventional ways. 

The  interpolation  method  requires  additional  explanation. 
Recall  that  to  lessen  the  burden  placed  on  the  participants, 
PSR's  were  developed  for  only  five  of  the  nine  combina- 
tions of  tree  size  classes  and  stand  density  classes.  A  pos- 
sible array  of  median  habitat  quality  results  is  illustrated: 

Stand  Tree  size  class 

density  class  S/S     Pole  Saw 

Poor  2        —  5 

Medium  —  5  — 
WeU  4        _  7 


We  felt  an  obligation  to  "fill-in-the-blanks."  Study  team 
member,  Robert  S.  Loveless,  Jr.,  developed  a  procedure  for 
doing  so.  The  procedure  could  be  called  "two-dimensional 
interpolation"  and  is  based  on  the  technique  of  Median 
Polish  (see  Velleman  and  HoagHn  1981  or  Tukey  1977). 
The  procedure  works  by  discovering  and  then  using  pat- 
terns within  arrays.  The  array  shown  above  has  pattern, 
ordered  from  left  to  right,  and  top  to  bottom.  The  missing 
values  are  numerically  between  or  equal  to  those  on  either 
side,  above,  or  below.  All  missing  cell  values  are  deter- 
mined by  multiplying  the  percentage  diagonal  change  by 
the  row  or  column  differences  and  then  adding  that  prod- 
uct to  a  leftmost  or  uppermost  known  value;  this  result  is 
rounded  to  the  nearest  integer.  Figui^e  3  illustrates  the  re- 
sult of  this  interpolation  process.  The  pattern  embedded  in 
the  original  data  provides  the  basis  for  specifying  the  over- 
all pattern. 

Two  final  points  regarding  analysis  should  be  mentioned. 
First,  statistical  tests  were  always  conducted  at  the  0.05 
level  of  significance.  Second,  the  number  of  observations 
used  in  specific  analyses  frequently  varied,  depending  on 
the  purpose  of  the  analysis.  When  necessary,  observations 
were  eliminated  from  analyses  to  achieve  balance  in  ex- 
perimental design;  balance  here  refers  to  a  situation  where 
there  is  an  equal  number  of  observations  in  each  analysis 
class  or  subclass. 

RESULTS 

During  August  1982  the  first  packet  of  materials  was 
sent  to  each  participant.  By  August  1983  all  data  had 
been  collected  for  summer  cover  and  summer  forage  and 
the  exit  interview  questionnaire  had  been  completed. 
Results  are  divided  into  two  sections— Elk  Habitat  Quality 
and  the  Delphi  Process. 

Elk  Habitat  Quality 

This  section  describes  the  results  of  elk  habitat  quality 
ratings  and  provides  results  from  three  major  types  of 
analyses  pertaining  to  those  ratings.  The  first  analysis 
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assessed  elk  habitat  quality  as  related  to  forest  habitat 
type  groups,  vegetation  cover  type,  tree  size,  and  stand 
density.  The  second  analysis  assessed  habitat  quality  from 
the  standpoint  of  the  underlying  components  of  forest  hab- 
itat type,  such  as  site  elevation  and  moisture.  The  third 
type  of  analysis  developed  a  tool  by  which  elk  habitat  set- 
tings can  be  classified  or  assigned  to  a  habitat  quality 
class  based  on  site  characteristics. 


Descriptive  Results.— The  final  measure  of  elk  habitat 
quality  is  the  median  rating  associated  with  the  third  iter- 
ation. Table  3  shows  these  medians  expressed  as  letter 
designations  (such  as  L  for  low),  along  with  inteipolated 
results  for  elk  summer  cover  and  forage.  For  both  cover 
and  forage,  results  for  the  171  settings  spanned  the  qual- 
ity range  from  very  low  to  very  high.  A  comparison  be- 
tween cover  and  forage  ratings,  for  an  individual  setting. 


Table  3.  — Final  elk  summer  cover  and  forage  quality  ratings,  by  vegetation  cover  type,  forest  fiabitat  type 
group,  stand  density  class,  and  tree  size  class 

 Tree  size  class   

Stand   Cover    Forage  

Vegetation  Forest  habitat      density       Seedling/    Pole-     Saw-      Seedling/    Pole-  Saw- 

cover  types  type  groups         class  sapling    timber  timber      sapling    timber  timber 


Hardwoods  Bottomlands  Poor 

Medium 
Well 

Spruce  Poor 

t^edium 

Well 

Ponderosa  pine        Ponderosa  pine  Poor 

tvledium 
Well 

Douglas-fir  I  Poor 

Medium 
Well 

Douglas-fir  II  Poor 

Medium 
Well 

Douglas-fir  III  Poor 

Medium 
Well 

Douglas-fir  V  Poor 

Medium 
Well 

Scree  Poor 

Medium 
Well 

Douglas-fir  Douglas-fir  I  Poor 

Medium 
Well 

Douglas-fir  II  Poor 

Medium 
Well 

Douglas-fir  III  Poor 

Medium 
Well 

Douglas-fir  IV  Poor 

Medium 
Well 

Douglas-fir  V  Poor 

Medium 
Well 

Subalpine  fir  1  Poor 

Medium 
Well 

Scree  Poor 

Medium 
Well 

Lodgepole  pine        Douglas-fir  III  Poor 

Medium 
Well 

Douglas-fir  IV  Poor 

Medium 
Well 

Subalpine  fir  I  Poor 

Medium 
Well 


1 

LM 

M 

H  M 

M 

IVI 

LM 

IVI 

HM 

M 

\A 
IVI 

M 

LM 

H  M 

H 

M 

M 

M 

1_IVI 

M 

\/H 
v  n 

u 
n 

LM 

IVI 

HM 

VH 

V  n 

H 

LM 

M 

H 

VH 

VH 

H 

VL 

L 

M 

l_ 

VL 

1 

L 

1 

l_ 

L 

L 

LM 

M 

L 

[_ 

L 

VL 

L 

LM 

l_ 

l_ 

LM 

|_ 

L 

LM 

M 
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[_ 
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LM 
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Table  3.  (Con.) 


 Tree  size  class  

Stand   Cover    Forage  

Vegetation  Forest  habitat        density     Seedling/    Pole-     Saw-      Seedling/    Pole-  Saw- 

cover  types  type  groups  class        sapling    timber  timber      sapling    timber  timber 


Spruce  Poor 

Medium 
Well 

Subaipine  fir  II  Poor 

Medium 
Well 

Subaipine  fir  III  Poor 

Medium 
Well 

Subaipine  fir  IV  Poor 

Medium 
Well 

Western  larcfi  Subaipine  fir  I  Poor 

Medium 
Well 

Engelmann  spruce      Subaipine  fir  I  Poor 

Medium 
Well 

Subaipine  fir  II  Poor 

Medium 
Well 

Subaipine  fir  IV  Poor 

Medium 
Well 

Spruce  Poor 

Medium 
Well 

Grand  fir  Western  redcedar/  Poor 

grand  fir  Medium 
Well 

Western  redcedar     Western  redcedar/  Poor 

grand  fir  Medium 
Well 

Subaipine  fir  Subaipine  fir  II  Poor 

Medium 
Well 

Subaipine  fir  III  Poor 

Medium 
Well 

Subaipine  fir  IV  Poor 

Medium 
Well 

Wfiitebark  pine        Subaipine  fir  III  Poor 

Medium 

Well 

Whitebark  pine  Poor 

Medium 

Weil 

Forbs  and  grasses     Mountain  meadow  — 
Mountain  grassland  — 
Shirubs  Mountain  brusfi  — 
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'Quality  ratings  in  small  print  were  developed  by  an  interpolation  process. 
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shows  both  consistency  and  great  divergence.  Not  unex- 
pected, the  mountain  meadow  setting  had  the  greatest  rat- 
ing differences— "very  low"  for  cover  and  "very  high"  for 
forage.  On  the  other  hand,  the  settings  containing  the 
scree  habitat  type  received  consistent  ratings  for  cover  and 
forage— consistently  low. 

Inspection  of  the  ratings  in  table  3  also  pro\ddes  a  pre- 
view of  the  analysis  results  presented  later.  Note  that 
while  the  cover  ratings  generally  increase  with  shifts  from 
poorly  stocked  seedhng-saphngs  to  weU-stocked  sawtimber, 
the  ratings  are  far  more  erratic  between  forest  habitat 
type  groups  or  vegetation  cover  types.  Similarly,  shifts  be 
tween  tree  size  and  stand  densitj^  classes  do  not  seem  to 
affect  forage  ratings,  while  cover  type  and  habitat  type 
gi'oups  do. 

A  general  description  of  habitat  ratings  would  not  be 
complete  without  some  indication  of  variation  in  the 
results.  Appendix  B  contains  a  complete  tabulation  of  vari- 
ation levels,  as  reflected  by  the  mean  deviation  from  the 
median  rating,  for  all  elk  habitat  settings  studied.  The 
average  of  the  mean  deviations  fi-om  the  medians  for  cover 
was  0.5  and  0.4  for  forage.  That  is.  forage  ratings  were 
about  20  percent  less  variable  than  were  cover  ratings.  Be- 
cause each  unit  of  deviation  represents  a  class,  the  average 
deviations  above  correspond  to  about  half  a  class  interval. 
The  most  variation  was  foimd  in  cover  ratings  for  several 
cases  involving  ponderosa  pine  or  Douglas-fir  cover  tj'pes 
where  the  mean  deviation  from  the  median  was  1.1  or 
more.  The  least  variation  (0.0)  existed  with  forage  ratings 
for  scree  and  mountain  meadow. 

Figure  4  shows  a  frequency  distribution  of  all  171  elk 
habitat  settings  in  each  habitat  quality  rating  class  for 
both  summer  cover  and  forage.  These  distributions  are 
visibly  "normal"  in  that  they  resemble  a  bell-shaped  curve. 
In  both  cases,  the  median  and  modal  class  is  "medium." 
While  the  middle  five  classes  show  a  pattern  of  con- 
sistency between  cover  and  forage,  the  extreme  classes 
(very  low  and  veiy  high)  show  opposite  patterns.  Rela- 
tively more  elk  habitat  settings  are  rated  "verj'  high"  for 
forage  than  for  cover  (17  versus  6);  and  relatively  more 
settings  are  rated  "very  low"  for  cover  than  for  forage  (16 
versus  10). 

Vegetation  Structure  and  Type.— This  line  of  analysis 
sought  to  determine  the  influence  of  the  primary  indepen- 
dent factors— cover  type,  tree  size,  and  so  forth— and  their 
interactions  on  elk  habitat  quality.  The  main  difficulty  was 


that  the  desired  analysis  could  not  be  performed  directly 
because  the  experimental  design  was  not  balanced:  that  is. 
we  did  not  have  an  equal  number  of  observations  associated 
with  all  combinations  of  levels  for  the  independent  varia- 
bles. Consequently,  we  had  to  conduct  three  analyses  that 
sequentially  led  to  the  final  results.  Desired  balance  was 
achieved  by  eliminating  observations.  Throughout  these 
analyses,  observ^ations  from  all  11  participants  were  used. 
The  largest  data  set  contained  1,881  (11  X  171)  observa- 
tions and  the  smallest  contained  1,452  observations. 

The  first  series  of  analyses  were  one-factor  analyses  of 
variance.  Their  purpose  was  to  determine  which  individual 
factor(s)  had  the  largest  and  smallest  effects  on  habitat 
quality  ratings.  All  1,881  observations  were  used.  Each 
independent  factor  or  factor  combination  was  used  in 
separate  analyses,  isolated  from  other  factors.  Table  4 
combines  the  results  and  shows  the  degrees  of  freedom. 


16 


23 


38 


47 


28 


13 


6 


10 


25 


31 


35 


29 


24 


1  7 


VL    L   LM  M    MH  H  VH 
COVER 


VL    L    LM  M   MH  H  VH 
FORAGE 


Figure  4.— Frequency  distribution  of  171  elk 
liabitat  settings,  by  final  habitat  quality 
ratings,  for  summer  cover  and  forage. 


Table  4.— One-factor  analysis  of  variance  showing  the  independent  influence 
of  six  factors  on  elk  summer  cover  and  forage  quality  ratings 


Source  of 
variation 


Degrees  of 
freedom 


Cover 


Forage 


SS 


SS 


Habitat  type  (HT)  17 

Cover  type  (CT)  11 

HT/CT  36 

Tree  size  (TS)  3 

Stand  density  (SD)  3 

TS/SD  5 

Error   

Total  1.880 


1,215     31.4  <0.001      4.791   468.0  <Q.001 


892 
1.295 

663 
2.260 
2.727 


33.2 
15.9 
86.5 
442.4 
374.6 


<0.001 

<  0.001 

<  0.001 

<  0.001 

<  0.001 


2.766 
4.809 
57 
94 
99 


149.3 
223.1 

6.1 
10.1 

6.4 


<0.001 
<0.001 
0.001 
<0.001 
<0.001 


 varies 

5.457     —  — 


5,913  — 
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sums  of  squares  (SS),  the  F-statistic  (F),  and  the  statisti- 
cal significance  level  (a)  associated  with  each  PSR  com- 
ponent pertaining  to  cover  or  forage.  Individually,  all 
factors  were  highly  significant.  Inspection  of  sums  of 
squares  can  indicate  relative  importance  in  accounting 
for  variation  in  cover  or  forage  ratings.  The  cover 
analysis  indicates  that  structure  of  vegetation— TS,  SD, 
TS/SD— is  far  more  important  to  cover  quality  ratings 
than  is  type  of  vegetation— habitat  type  group  (HT), 
cover  type  (CT),  HT/CT.  Of  the  two  structure-oriented 
variables  (TS  and  SD)  the  stand  density  variable  (SD)  is 
more  significant,  but  tree  size  (TS)  is  also  important.  In 
contrast,  the  forage  analysis  shows  that  type  of  vegeta- 
tion is  more  important  to  forage  quality  ratings  than  is 
structure.  Furthermore,  habitat  type  group  (HT)  alone  is 
more  important  than  cover  type  (CT)  alone,  explaining 
almost  as  much  variation  as  the  combination,  HT/CT. 
But  the  problem  with  one-factor  analyses  is  that  they 
are  incapable  of  assessing  interaction  of  factors. 

The  second  series  of  analyses  were  two-factor  analyses 
of  variance  on  cover  and  forage  ratings.  The  data  set 
needed  to  be  reduced  to  1,815  observations  to  achieve 
statistical  balance.  The  two  factors  considered  were  the 
combinations  of  HT/CT  and  TS/SD.  The  purpose  was  to 
assess  the  role  of  the  overall  interaction  between  vegeta- 
tion structure  and  vegetation  type  in  habitat  quality. 
Table  5  shows  the  results,  reinforcing  the  importance  of 
vegetation  structure  (TS/SD)  for  cover  and  vegetation 


type  (HT/CT)  for  forage.  But  these  analyses  were  per- 
formed to  examine  the  interaction  of  vegetation  struc- 
ture and  type.  F-tests  of  the  interaction  between  vegeta- 
tion type  and  structure  (AXB)  were  statistically 
significant  for  both  cover  and  forage.  However,  that 
statistical  significance  clearly  resulted  from  the  enor- 
mous degrees  of  freedom  in  the  error  term.  We  judged 
the  magnitude  of  the  interaction  effect,  as  indicated  by 
sums  of  squares,  to  be  trivial— too  small  for  practical  im- 
portance. We  therefore  elected  to  ignore  interaction 
effects  between  structure  (TS/SD)  and  type  (HT/CT)  of 
vegetation. 

The  third  series  of  analyses  were  designed  to  assess 
the  more  restricted  interaction  of  tree  size  (TS)  or  stand 
density  (SD)  with  type  of  vegetation  (HT/CT).  The  three- 
factor  analysis  of  variance  used  again  called  for  a  reduc- 
tion in  the  number  of  observations  in  order  to  achieve 
statistical  balance.  Table  6  contains  the  results.  The  pat- 
tern of  statistically  significant  interaction  terms  is  more 
pronounced  in  the  cover  than  forage  analysis.  In  fact, 
for  forage,  three  of  the  four  interaction  terms  were  not 
statistically  significant.  The  most  substantial  interaction 
found  was  between  vegetation  type  (HT/CT)  and  tree 
size  (TS)  regarding  cover  ratings.  Nevertheless,  even  for 
the  significant  terms,  the  magnitude  of  the  interaction 
effects  shown  by  their  sums  of  squares  was  trivially 
small  and  overwhelmed  by  the  main  effects.  Interaction 
effects  were  again  ignored. 


Table  5.— Two-factor  analysis  of  variance  showing  the  influence  of  HT/CT, 

TS/SD,  and  their  interaction  on  elk  summer  cover  and  forage  habitat 
quality  ratings 


Source  of 
variation 

Degrees  of 
freedom 

Cover 

Forage 

SS 

F 

SS 

F  a 

HT/CT  (A) 

32 

1,069 

43.8 

<0.001 

4,660 

270.3  <0.001 

TS/SD  (B) 

4 

2,524 

828.3 

<0.001 

62 

128.8  <0.001 

Interaction:  A  ■  B 

128 

298 

3.1 

<  0.001 

107 

1.6  <0.001 

Error 

1,650 

1.258 

889 

Total 

1,814 

5,150 

5,718 

Table  6.— Three-factor  analysis  of  variance  showing  the  influence  of  HT/CT,  TS,  SD,  and  their 
interactions  on  elk  summer  cover  and  forage  quality  ratings 


Source  of 
variation 

Degrees  of 
freedom 

Cover 

Forage 

SS 

F 

a 

SS 

F 

a 

HT/CT  (A) 

32 

759.0 

29.6 

<  0.001 

3,646.00 

212.9 

<0.001 

TS  (B) 

1 

439.0 

547.0 

<  0.001 

5.00 

9.5 

0.003 

SD  (C) 

1 

1,976.0 

2,464.9 

<  0.001 

54.00 

100.9 

<  0.001 

Interactions;  A  «  B 

32 

107.0 

134.0 

<0.001 

24.00 

1.4 

ns^ 

A  X  C 

32 

81.0 

101.5 

<  0.001 

40.00 

2.3 

<  0.001 

BxC 

1 

6.9 

8.6 

0.004 

.04 

.1 

ns 

AxBxC 

32 

50.6 

63.1 

<0.001 

25,00 

1.4 

ns 

Error 

1,320 

1,058.0 

706.00 

Total 

1,451 

4,479.0 

4,500.00 

'Statistically  nonsignificant  at  0.05  level. 
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Table  7.  — Final  analysis  of  variance  showing  influence  of  various  factors  and  tfieir  interactions 
on  elk  summer  cover  and  forage  quality  ratings 


Source  of 
variation 

Degrees  of 
freedom 

Cover 

Forage 

SS 

F  a 

SS 

F 

a 

Habitat  type  (HT) 

17 

not  included  

4,791 

468.0 

<  0.001 

Cover  type  (CT) 

11 

not  included  

not  included 

HT/CT 

36 

1,295 

15.9  <0.001 

not  included 

Tree  size  (TS)^ 

3 

663 

86.5  <0.001 

not  included 

Stand  density  (SD)^ 

3 

2,260 

753.5    <  0.001 

94 

10.1 

<  0.001 

TS/SD 

5 

not  included  

not  included 

Interactions 

varies 

not  included  

not  included 

Error 

varies 

1,239 

1.028 

Total 

1,881 

5,457 

5.913 

^Nonstocked  was  also  considered  a  size  and  density  class  in  these  analyses. 


Table  8.  — Ranked  effect  of  forest  fiabitat  type  group/vegetation  cover  type  combinations,  tree  size  class,  and 
stand  density  class  on  elk  summer  cover  quality  ratings,  for  final  modeP 


 Combinations  

Cover  types  Habitat  type  groups         Contribution    Tree  size  class  Contribution 


Subalpine  fir 

Subalpine  fir  1 

1.6 

Sawtimber  0.4 

Western  redcedar 

Western  redcedar/grand  fir 

1.6 

Poletimber  .4 

Engelmann  spruce 

Subalpine  fir  II 

1.3 

Seedling/sapling  -.7 

Grand  fir 

Western  redcedar/grand  fir 

1.2 

Lodgepole  pine 

Subalpine  fir  II 

1.1 

Engelmann  spruce 

Subalpine  fir  IV 

1.0 

Stand  density  class 

Engelmann  spruce 

Spruce 

.8 

Engelmann  spruce 

Subalpine  fir  II 

.8 

Well  1.1 

Subalpine  fir 

Subalpine  fir  IV 

.7 

Medium  .4 

Hardwoods 

Spruce 

.4 

Poor  -1.3 

Western  larch 

Subalpine  fir  1 

.3 

Lodgepole  pine 

Subalpine  fir  1 

.3 

Douglas-fir 

Douglas-fir  IV 

.2 

Douglas-fir 

Subalpine  fir  1 

.2 

Ponderosa  pine 

Douglas-fir  II 

.2 

Hardwoods 

Bottomlands 

.2 

Lodgepole  pine 

Spruce 

.1 

Subalpine  fir 

Subalpine  fir  III 

0 

Douglas-fir 

Douglas-fir  II 

0 

Lodgepole  pine 

Subalpine  fir  IV 

-.1 

Lodgepole  pine 

Douglas-fir  IV 

-.2 

Ponderosa  pine 

Douglas-fir  III 

-.3 

Lodgepole  pine 

Douglas-fir  III 

-.3 

Shrubs 

Mountain  brush^ 

-.3 

Douglas-fir 

Douglas-fir  III 

■  -.3 

Whitebark  pine 

Subalpine  fir  III 

-.4 

Lodgepole  pine 

Subalpine  fir  III 

-.4 

Whitebark  pine 

Whitebark  pine 

-.6 

Douglas-fir 

Douglas-fir  V 

-.6 

Ponderosa  pine 

Douglas-fir  V 

-.7 

Douglas-fir 

Douglas-fir  1 

-1.0 

Ponderosa  pine 

Douglas-fir  1 

-1.0 

Ponderosa  pine 

Ponderosa  pine 

-1.1 

Douglas-fir 

Scree 

-1.4 

Ponderosa  pine 

Scree 

-1.7 

Forbs  and  grasses 

Mountain  meadow^ 

-2.5 

Forbs  and  grasses 

Mountain  grassland^ 

-2.5 

^Overall  mean  habitat  rating  =  3.6. 

^Tree  size  class  and  stand  density  class  contributions  are  not  applicable. 
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Final  judgments  on  the  importance  of  various  factors 
and  their  interactions  were  based  on  the  knowledge  gained 
in  the  one-factor,  two-factor,  and  three-factor  analyses. 
Statistical  significance  and  simplicity  were  important 
considerations.  A  simpler  model  was  selected  over  a 
more  complex  one  if  both  were  roughly  equivalent  in 
helping  understand  habitat  quality.  For  example,  while 
interaction  terms  were  frequently  found  significant,  their 
contribution  to  understanding  habitat  quality  was  small, 
but  their  addition  to  complexity  was  great.  The  three- 
factor  model,  without  interactions,  was  selected  as  the 
best  model  for  understanding  summer  cover  quality. 
Summary  analysis  of  variance  results  (table  7)  shows 
that  the  sums  of  squares  for  stand  density  are  clearly 
the  most  influential  factor  in  explaining  cover  quality 
ratings,  followed  by  the  combination  of  habitat  type 
group  with  cover  type,  and  finally,  tree  size  class.  These 
factors  account  for  77.3  percent  of  the  variation  in  cover 
quality.  A  two-factor  model,  consisting  only  of  habitat 
type  groups  and  stand  density,  was  selected  for  under- 
standing forage  quality.  In  forage  analyses  containing 
interaction  terms,  the  interactions  were  either  statisti- 
cally insignificant  or  so  small  they  could  be  ignored.  Fur- 
thermore, almost  all  variation  explained  by  the  combina- 
tion TS/SD  was  accounted  for  by  SD  alone.  Similarly, 
HT  accounted  for  nearly  all  of  the  variation  explained  by 
the  combination  of  HT/CT.  Table  7  also  displays  analysis 
of  variance  results  for  forage.  The  factors  shown  accounted 
for  82.6  percent  of  the  vajiation  in  forage  quality  ratings. 

The  cover  model  in  table  7  can  be  presented  in  equa- 
tion form  as  a  three-factor  statistical  model  without 
interaction  terms: 

C,j^  -  ^^  +  (HT/CT),  +  TS^  +  SD, 

This  equation  is  interpreted  as  follows:  the  cover  quality 
(C)  of  a  specific  elk  habitat  setting  (ijk)  is  equal  to  the 


overall  cover  mean  (n),  plus  the  contribution  associated 
with  its  specific  combination  (i)  of  habitat  type  group 
and  cover  type  (HT/CT),  plus  the  contribution  of  its 
specific  (j)  tree  size  class  (TS),  plus  the  contribution  of 
its  specific  (k)  stand  density  class  (SD).  The  overall  cover 
mean  (/*)  for  all  elk  habitat  settings  is  3.6.  Table  8  shows 
the  contributions  associated  with  levels  of  all  factors. 
Both  the  sign  and  magnitude  of  the  contributions  are 
meaningful.  Consider  the  sign:  if  the  sign  of  the  contri- 
bution is  positive,  the  cover  value  is  greater  than  the 
overall  mean;  if  negative,  it  is  less.  The  first  17  combina- 
tions of  HT/CT  have  positive  contributions,  the 
remainder  are  zero  or  have  negative  signs.  Similarly,  two 
levels  of  the  tree  size  and  stand  density  are  positive,  the 
other  is  negative.  Poorly  stocked  stands  or  seedling- 
sapling  trees  tend  to  force  the  overall  rating  below  the 
mean.  The  magnitude  of  the  contributions  is  also  impor- 
tant. The  larger  the  contribution,  the  greater  the  effect, 
positive  or  negative.  A  contribution  of  1.0  will  have 
twice  the  effect  on  cover  quality  as  will  a  value  of  0.5.  A 
well-stocked  stand  will  tend  to  be  rated  at  least  two 
classes  higher  (1.1  +  1.3)  than  an  identical  situation,  but 
under  poor  stocking. 

The  forage  model  reflected  in  table  7  can  also  be 
expressed  as  a  two-factor  statistical  model  without  inter- 
action terms: 

F„  =  /(  +  HT_  +  SD, 

Here,  the  forage  value  (F)  of  a  specific  elk  habitat  set- 
ting (ij)  is  equal  to  the  overall  forage  mean  (/<),  plus  the 
contribution  of  a  specific  (i)  habitat  type  group  (HT), 
plus  the  contribution  of  a  specific  (j)  stand  density  (SD) 
class.  Table  9  contains  these  contributions.  The  overall 
forage  mean  is  4.1.  All  interpretations  regarding  sign 
and  magnitude  of  contributions  pertaining  to  forage  are 
the  same  as  discussed  previously  for  cover. 


Table  9.  — Ranked  effects  of  forest  habitat  type  groups  and  stand  density  class  on 

elk  summer  forage  quality  ratings,  for  final  modeP 

Habitat  type  groups 

Contribution 

Stand  density  class  Contribution 

Mountain  meadow^ 

2.9 

Poor  0.2 

Spruce 

2.4 

Medium  .1 

Subalpine  fir  1 

2.2 

Well  -.2 

Western  redcedar/grand  fir 

1.3 

Subalpine  fir  II 

1.1 

Subalpine  fir  IV 

.5 

Mountain  grassland^ 

.4 

Mountain  brush^ 

.3 

Bottomland 

.1 

Douglas-fir  IV 

.    -  .2 

Douglas-fir  III 

-  .6 

Whitebark  pine 

-  .6 

Douglas-fir  II 

-1.0 

Subalpine  fir  III 

-  1.0 

Douglas-fir  V 

-  1.7 

Ponderosa  pine 

-  1.9 

Douglas-fir  1 

-2.0 

Scree 

-3.1 

'Overall  mean  habitat  rating  =  4.1. 

^Stand  density  class  contributions  are  not  applicable. 
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Habitat  Type  Components.— For  some  purposes, 
assessing  elk  habitat  quality  in  terms  of  vegetation  type 
and  structure  may  be  adequate.  For  other  purposes  it  is 
desirable  to  better  understand  why  habitat  quality 
varies.  In  this  regard,  the  concept  of  forest  habitat  type 
is  a  key  consideration  because  it  is  a  composite  of 
several  environmental  factors.  In  short,  while  habitat 
type  per  se  is  a  taxonomic  structure,  it  consists  of 
environmental  factors  that  actually  cause  variation  in 
habitat  quality. 

The  study  team  identified  six  major  components  of 
forest  habitat  type:  type  of  understory  vegetation,  den- 
sity of  understory  vegetation,  elevation,  topographic  set- 
ting, moisture,  and  temperature.  Each  component  was 
subdivided.  For  example,  temperature  was  divided  into 
hot,  warm,  moderate,  and  cool.  Each  of  the  171  elk  habi- 
tat settings  was  coded  according  to  these  components 
and  subdivisions.  A  series  of  analj'ses  were  conducted  to 
determine  if  the  categories  identified  were  statistically 
distinguishable  from  each  other  in  terms  of  habitat  qual- 
ity ratings.  Frequently  they  were  not.  When  insignifi- 


cant differences  were  detected,  subdivisions  were  recom- 
bined  into  the  following  divisions: 

Component  Divisions 

Understory  type         Primarily  grasses  (UTl) 

Primarily  shrubs  (UT2) 
Understory  density     Sparse  or  moderate  (UDl) 

Dense  or  varied  (UD2) 
Elevation  Low  or  middle  (El) 

High  or  aU  (E2) 
Topograpic  setting     South  or  west  face  (TSl) 

All  faces  (TS2) 

Bottomland  or  benches  (TS3) 
Moisture  Dry  or  moderately  dry  (Ml) 

Moderate  or  wet  (M2) 
Temperature  Hot  or  warm  (Tl) 

Moderate  or  cool  (T2) 

Figure  5  shows  the  percentage  distribution  of  habitat 
quality  ratings  over  all  habitat  tyipe  components  and 
their  associated  divisions,  for  both  cover  and  forage.  For 
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Figure  5.— Percentage  distribution  of  171  eik  tiabitat  settings,  by  final  hiabitat  quality  ratings, 
for  components  of  forest  habitat  type,  summer  cover,  and  forage. 
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these  components  to  be  useful  in  understanding  elk  habi- 
tat quality,  trends  should  be  present.  Perhaps  the  most 
obvious  trend  pertains  to  site  moisture  and  forage  qual- 
ity ratings.  There,  100  percent  of  the  PSR's  rated  very 
low  or  low  (1  or  2)  were  from  dry  or  moderately  dry 
sites;  conversely,  100  percent  of  the  PSR's  rated  high  or 
very  high  (6  or  7)  were  from  moderate  or  wet  sites. 

A  series  of  one-factor  analyses  were  conducted  to 
determine  which  single  component  of  forest  habitat  type 
had  the  greatest  effect  on  cover  and  forage  quality. 
Table  10  shows  these  analyses.  Because  most  compo- 
nents were  highly  significant,  attention  was  focused  on 
the  sums  of  squares  for  further  interpretation.  Not  sur- 
prisingly, habitat  type  components  had  little  influence 
on  cover  quality  ratings,  the  largest  sums  of  squares 
accounting  for  only  61  of  the  378  total.  The  same  rela- 
tionship was  shown  earlier  in  table  4  where  the  habitat 
type  factor  (not  the  components)  was  shown  to  be 
strongly  related  to  forage  but  not  to  cover  quality.  Fur- 
ther examination  revealed  that  the  most  important  com- 
ponents for  cover  were  topographic  setting,  density  of 
understory  vegetation,  and  moisture.  The  most  impor- 
tant components  for  forage  were  topographic  setting, 
moisture,  and  density  of  understory  vegetation. 

Subsequent  analysis  of  habitat  type  components  was 
thwarted  for  two  reasons.  First,  because  these  compo- 
nents were  not  the  primary  independent  variables  used 
in  this  study,  they  played  no  role  in  the  basic  experimen- 


tal design.  When  the  set  of  elk  habitat  settings  was 
selected  to  approximate  a  balanced  experimental  design, 
habitat  type  components  were  not  considered  as  part  of 
the  balance.  Consequently,  multiple  linear  regression 
analysis  was  appropriate  for  use.'  Second,  habitat  type 
components  are  not  independent  of  each  other.  For 
example,  wetter  sites  tend  to  be  cooler  sites  and  drier 
sites  tend  to  be  warmer  sites.  Because  of  the  problem  of 
nonindependence  between  components,  regression  results 
should  be  interpreted  as  indicative,  not  conclusive.  The 
results  are  useful  only  in  accounting  for  variation  in  hab- 
itat quality  ratings,  not  in  understanding  the  underlying 
biological  causes  and  effects. 

A  multiple  linear  regression  model  was  developed  for 
both  cover  and  forage  quality.  The  levels  associated  with 
habitat  type  components  were  handled  as  "dummy  vari- 
ables." Table  11  shows  the  results  of  these  analyses.  For 
the  forage  model,  87  percent  of  the  total  variation  in 
habitat  quality  ratings  was  explained  by  the  six  statisti- 
cally significant  habitat  type  components.  Only  about  19 
percent  of  the  total  variation  in  cover  ratings  was 
explained  by  the  two  statistically  significant  compo- 
nents. This  is  not  surprising  because  we  have  already 


'When  an  analysis  involves  an  unequal  number  of  observations  in  a 
subclass,  routine  analysis  of  variance  techniques  cannot  be  used  to 
determine  the  influence  of  various  factors  on  a  dependent  variable.  In 
such  cases,  multiple  linear  regression  techniques  must  be  used  (Draper 
and  Smith  1981). 


Table  10.— One-factor  analysis  of  variance  showing  the  independent  influence  of  six  habitat  type 
components  on  elk  summer  cover  and  forage  quality  ratings 


Source  of  Degrees  of   Cover    Forage 

variation  freedom  SS  F  a  SS  F 


Understory  density  1  43.6  22.1  <0.001  172.7  104.2  <0.001 

Understory  type  1  31.1  15.1  <0.001  88.2  40.9  <0.001 

Elevation  1  1.8           .8  ns^  10.9  4.2  .039 

Topographic  setting  2  61.2  16.2  <0.001  327.3  219.4  <0.001 

Moisture  1  37.5  18.6  <0.001  251.9  212.1  <0.001 

Temperature  1  12.4          5.7  .017  83.1  38.0  <0.001 

Error    varies  

Total  170  377.8  —  —  452.6  —  — 


'Statistically  nonsignificant  at  0.05  level. 


Table  11.  — Linear  regression  models  and  analysis  of  variance  showing  the  influence  of 
several  independent  factors  (habitat  type  components)  on  elk  summer  cover 
and  forage  quality  ratings 


MODEL: 

Cover 

Forage 

C  = 

4.18  -1.11{TS1)  - 

0.576(M1) 

F  = 

6.5  -1.35{M1)  -2.8(TS1)  -1.3(TS2) 

-0.87(T1)  -0.62(E1)  +0.72(UT1) 

ANOVA: 

Source 

df 

SS  F 

a 

df 

SS                    F  a 

Regression 

2 

70.2  19.2 

<  0.001 

6 

394.4         185.1  <0.001 

Error 

168 

307.6  — 

164 

58.2            —  — 

Total 

170 

377.8  — 

170 

452.6            —  — 

R2  =  0.19  R2  =  0.87 
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shown  cover  to  be  closely  associated  with  stand  struc- 
ture rather  than  habitat  type.  In  both  models,  the  topo- 
graphic setting  and  moisture  were  statistically  signifi- 
cant variables.  Additionally,  temperature,  elevation,  and 
the  type  of  understory  vegetation  were  statistically  sig- 
nificant in  the  forage  model. 

Table  12  shows  the  effect  of  the  statistically  signifi- 
cant independent  variables  on  cover  or  forage  quality 
rating.  The  component  levels  shown  with  "  +  "  indicate 
that  when  that  level  is  present,  it  tends  to  increase  the 
overall  habitat  quality  rating;  a  "  — "  means  the  rating 
tends  to  be  decreased.  The  statistically  nonsignificant 
(nss)  component  levels  mean  that  their  effect  on  habitat 
quality  is  so  variable  (erratic)  that  no  additional  sys- 
tematic positive  or  negative  effect  on  habitat  quality 
could  be  discerned,  given  the  presence  of  the  statistically 
significant  variables  already  in  the  model.  Consider  for- 
age quality.  Sites  consisting  primarily  of  a  grass  under- 
story have  a  statistically  significant  higher  quality  than 
those  consisting  primarily  of  shrubs.  Similarly,  under- 
story density  does  not  have  a  statistically  significant 
effect  on  forage  quality  after  moisture,  temperature,  and 
other  habitat  type  components  have  been  used  to 
explain  rating  variability. 

Classification  Functions.— One  of  the  most  desired  out- 
comes of  our  study  was  the  development  of  a  field- 
oriented  tool  to  predict  habitat  quality.  The  typical  tool 
for  this  purpose  would  be  a  multiple  linear  regression 
model.  But  a  regression  model  requires  the  dependent 
variable,  habitat  quality,  to  be  measured  on  an  interval 
or  ratio  scale.  Our  habitat  quality  ratings  were  measured 
on  an  ordinal  scale.  However,  another  tool,  discriminant 
analysis,  was  appropriate. 

In  discriminant  analysis,  the  objective  is  to  assign  or 
classify  a  subject  into  one  or  more  groups  (the  depen- 
dent variable),  based  on  measurements  taken  on  a  set  of 
independent  variables.  In  our  case,  the  "subject"  cor- 
responds to  elk  habitat  settings  and  the  "groups"  cor- 
respond to  the  seven  habitat  quality  classes.  A  typical 


discriminant  analysis  develops  linear  classification  func- 
tions, one  for  each  group  associated  with  the  dependent 
variable  (seven  in  our  case).  The  subject  is  then  classi- 
fied into  the  group  based  on  the  highest  classification 
function  score.  The  approach  we  took  was  to  develop  a 
sequential  analysis  procedure  resembling  a  dichotomous 
key  used  in  taxonomy. 

Two  general  classes  of  independent  variables  were 
used:  habitat  type  components  (elevation,  moisture,  and 
so  forth)  and  combinations  of  tree  size  class  and  stand 
density  class  (TS/SD).  Habitat  type  component  codes 
and  groupings  are  the  same  as  before.  The  combinations 
of  TS/SD  are  coded  as  follows: 

Cl  —  Seedling  and  sapling,  poorly  stocked 

C2  —  Seedling  and  sapling,  well  stocked 

C3  —  Medium  stocked,  poletimber 

C4  —  Sawtimber,  poorly  stocked 

C5  —  Sawtimber,  well  stocked 

C6  —  Nonstocked. 

The  sequential  discriminant  analysis  allows  a  user  to 
apply  a  procedure  that  systematically  interprets  an  elk 
habitat  setting  until  the  "most  Ukely"  habitat  quality 
rating  has  been  assigned.  Figure  6  displays  this  sequen- 
tial procedure  in  the  form  of  a  dichotomous  key.  Its  use 
can  be  illustrated  with  an  elk  habitat  setting  with  the 
following  characteristics:  UTl,  UDl,  El,  TSl,  Ml,  Tl, 
and  C4.  All  variables  are  binary  (zero  or  one)  and  should 
be  interpreted  as  discussed  with  previous  regression 
analyses.  If  the  description  for  the  variable  is  present, 
the  variable  takes  on  a  value  of  one;  otherwise  the  value 
is  zero.-  Using  the  equation  associated  with  the  first 


-As  with  other  statistical  analysis  procedures,  discriminant  analysis 
typically  rests  on  a  set  of  underlying  assumptions.  Perhaps  the  most 
critical  is  that  of  multivariate  normality  of  the  independent  variable 
(Kleinbaum  and  Kupper  1978).  Our  analyses  violate  this  assumption 
because  all  independent  variables  are  discrete,  handled  as  binary, 
dummy  variables.  This  violation  precludes  rigorous  statistical  interpreta- 
tion of  the  discriminant  functions.  It  also  means  that  the  functions 
developed  may  not  be  optimal  (Goldstein  and  Dillon  1978).  However,  our 
use  of  the  functions  is  solely  for  classification  purposes,  and  we  make  no 
claim  regarding  optimality. 


Table  12.— Effect  of  habitat  type  components  on  elk  summer  cover  and 
forage  quality  ratings 


Component  level 

Cover 

Forage 

Understory  type  (UT) 

—  Primarily  grasses 

nss^ 

-1- 

—  Primarily  shrubs 

nss 

Understory  density  (UD) 

—  Sparse  or  moderate 

nss 

nss 

—  Dense  or  varied 

nss 

nss 

Elevation  (E) 

—  Low  or  middle 

nss 

—  High  or  all 

nss 

-1- 

Topographic  setting  (TS) 

—  South  or  west  face 

—  All  faces 

nss 

—  Bottomland  or  benches 

nss 

+ 

Moisture  (M) 

—  Dry  or  moderately  dry 

—  Moderate  or  wet 

-1- 

-1- 

Temperature  (T) 

—  Hot  or  warm 

nss 

—  Moderate  or  cool 

nss 

-1- 

^Not  statistically  significant  at  0.05  level. 
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branch  of  the  cover  dichotomous  key,  calculate  the  score: 

VL  =  -4.20  +8.07(0)  +1.95(1)  -1.51(1)  +2.22(1)  -1.59(1) 
=  -3.13 

Following  the  instructions,  the  magnitude  of  the  score 
(less  than  zero)  indicates  that  this  elk  habitat  setting  is 
"otherwise,"  not  "very  low,"  and  that  computations 
should  continue.  The  score  for  the  second-step  equation 
is  +4.12.  Because  this  score  exceeds  zero,  the  site  would 
be  assigned  a  "low"  habitat  quality  rating. 

The  reliability  of  a  discriminant  analysis  is  indicated 
by  its  ability  to  correctly  classify  subjects— elk  habitat 
settings.  Overall,  the  sequential  discriminant  function 
procedure  correctly  classified  60  percent  of  the  cover 
ratings  and  67  percent  of  the  forage  ratings.  Most  mis- 
classifications  occurred  within  a  more  general  rating, 
within  the  lows,  mediums,  and  highs.  For  example,  while 
the  procedure  had  difficulty  distinguishing  between  high 
and  very  high,  it  correctly  classified  all  of  the  "high" 
quality  sites.  We  repeated  the  sequential  analysis  using 
three  habitat  quaUty  classes  (low  =  very  low  and  low; 
medium  =  low  medium,  medium,  and  high  medium;  high 
=  high  and  very  high).  This  approach  correctly  classified 
84  percent  of  the  sites  for  cover  and  92  percent  for  for- 
age. The  dichotomous  key  for  the  three-habitat  quality 
class  analysis  is  shown  in  appendix  C. 

Delphi  Process 

Although  the  Delphi  process  is  rather  straightforward, 
there  are  several  major  decisions  that  must  be  made  and 
can  affect  results.  This  section  evaluates  a  wide  range  of 
study  results  pertaining  to  four  major  aspects  of  Delphi: 
(1)  selection  and  number  of  participants,  (2)  consensus 
achieved  and  what  affected  it,  (3)  the  data-gathering 
instrument  or  questionnaire  used,  and  (4)  time  considera- 
tions such  as  how  long  parts  of  the  Delphi  process  took. 

Participant  Selection  and  Number.— The  participant 
identification  process  began  with  the  "starter"  identify- 
ing eight  persons  as  experts  in  elk  habitat  for  western 
Montana.  If  any  of  several  other  ways  of  identifying 
experts  had  been  used  the  number  of  choices  would  have 
changed: 

Selection  method  Number  of  experts  identified 

Starter  only  8 

Any  one  identifier  2  —  19 

All  identifiers  combined  30 

Top-one  method  2 

Top-two  method  5 

Top-three  method  13 

Top-four  method  15 

Some  30  experts  were  identified  by  all  persons  con- 
tacted. If  any  one  person  had  been  contacted  as  our  sole 
source  of  "expert"  names,  as  few  as  two  or  as  many  as 
19  persons  would  have  been  identified  as  expert.  The 
number  of  experts  identified  by  the  "top"  method 
quickly  increases  as  the  top-three  method  is  approached 
and  then  levels  off.  We  suspect  that  if  a  top-five  method 
had  been  used,  results  would  have  been  essentially  the 
same  as  the  top-four  method. 

A  wide  difference  exists  between  individuals  in  their 
personal  perception  of  who  is  an  expert.  That  is  why  the 


top-three  format  identified  13  individuals,  not  three. 
Comparing  individuals  contacted,  perceptions  ranged 
from  total  agreement  to  total  disagreement  as  to  which 
persons  constitute  the  top  three  experts.  There  was 
much  more  consistency  in  identif\dng  experts  in  the  top- 
one  or  top-two  formats.  Figure  7  illustrates  a  typical 
pattern  in  group  dynamics,  focusing  on  five  individuals 
being  identified  as  in  the  top-two  format:  the  starter 
identified  A  and  B;  when  contacted.  A  and  B  identified 
two  new  top-two  experts  (C  and  D):  when  C  and  D  were 
contacted,  only  one  new  expert  (E)  was  identified;  when 
contacted,  E  identified  two  previously  identified  persons. 

The  participant  selection  method  should  be  related  to 
the  number  of  participants  desired.  If  a  small  number  of 
participants  is  adequate,  the  top-one  or  top-two  method 
might  be  appropriate.  If  a  larger  number  is  desired,  the 
top-three,  top-four,  or  all  combined  methods  might  have 
to  be  used.  This  study  used  all  approaches  because  we 
had  no  idea  how  many  experts  would  be  identified  by 
each  method.  We  wanted  a  large  group  so  that  we  could 
later  simulate  the  outcome  of  using  a  smaller  number  of 
participants.  Once  the  results  were  known,  we  selected 
the  13  experts  identified  by  the  top-three  method  to 
serve  as  participants  in  our  Delphi  project. 

Is  the  process  of  identifying  elk  habitat  experts 
affected  by  the  occupational  status,  degree  of  expertise, 
or  the  amount  of  professional  elk  habitat  experience  on 
the  part  of  those  persons  doing  the  identifying?  To 
address  this  issue,  each  of  the  13  experts  was  assigned  a 
code  reflecting  occupational  status,  expertise,  and 
experience.  Experts  were  assigned  to  an  occupational 


STARTER 


Figure  7.— Illustration  of  networking 
associated  with  a  top-two  procedure  of  expert 
identification. 
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category  based  on  their  current  job— either 
managers/administrators  or  researchers/academics.  Each 
expert  was  assigned  to  the  upper  or  lower  half  of  the 
expertise  listing,  based  on  the  frequency  of  being  identi- 
fied as  an  expert.  For  example,  based  on  the  pattern  of 
identification  shown  in  figure  7,  "B"  (named  six  times) 
would  be  assigned  to  the  upper  half  and  "E"  (named 
once)  to  the  lower  half.  A  measure  reflecting  experience 
was  developed.  Each  expert's  professional  career  was 
converted  to  an  expression  of  "annual  equivalents"  of 
elk  habitat  experience.  For  example,  10  years  of  involve- 
ment at  a  50  percent  level  of  commitment  was  expressed 
as  5  years  (or  annual  equivalents)  of  experience. 

Does  the  occupation,  expertise,  or  experience  of  an 
identifier  affect  the  number  of  persons  identified  as 
experts?  No.  A  series  of  one-factor  analyses  assessed 
whether,  for  example,  the  average  total  number  of 
experts  identified  by  managers/administrators  (M/A)  was 
significantly  different  from  the  average  identified  by 
researchers/academics  (R/A).  Table  13  shows  that,  over- 
all, each  person  identified  an  average  of  7.1  experts  and 
that  the  M/A  average  was  not  significantly  different 
from  the  R/A  average.  Similarly,  simple  linear  regression 
analysis  showed  that  the  years  of  experience  had  no 
statistically  significant  effect  on  the  total  number  of 
experts  identified.  In  fact,  we  found  no  evidence  that 
occupation,  expertise,  or  experience  had  any  effect  on 
the  number  of  experts  identified. 

Was  there  an  occupational  bias  in  the  type  of  expert 
identified?  That  is,  do  identifiers  from  one  occupational 
status  tend  to  identify  only  others  from  that  status  as 
experts?  Again  no.  A  chi-square  test  of  independence 


between  occupational  status  of  identifier  and  occupa- 
tional status  of  the  identified  experts  failed  to  detect  a 
statistical  dependency.  The  identifiers  were  not  occupa- 
tionally  biased  in  their  judgments. 

Although  the  top-three  method  identified  13  experts, 
only  11  actually  participated  throughout  the  study.  Two 
declined  participation  in  the  study  for  reasons  unrelated 
to  elk  habitat  quality  ratings.  The  11  who  participated 
were  distributed  between  occupational  status  and  exper- 
tise levels  as  follows: 

Occupation  status 
Level  of  expertise     M/A  R/A 
Upper  half  2  4 

Lower  half  J_ 

6  5 

After  all  elk  habitat  ratings  had  been  obtained,  the 
question  arose:  Are  the  ratings  related  to  the  profes- 
sional status  of  the  participants?  A  series  of  one-factor 
analyses  of  variance  were  conducted.  The  dependent 
variable  consisted  of  the  final  summer  cover  and  forage 
quality  ratings.  As  before,  the  independent  factors  of 
expertise  and  occupation  were  handled  as  having  two 
levels  each:  M/A  and  R/A  for  occupation  and  upper  and 
lower  half  for  expertise.  Experience  was  also  treated  as  a 
factor,  but  with  three  levels.  Because  participants  were 
divided  almost  evenly  into  three  experience  classes,  we 
labeled  the  classes  as  most,  least,  and  middle.  Table  14 
shows  the  resulting  analyses  of  variance.  Two  points 
warrant  mention.  First,  except  for  the  experience  factor 
in  the  forage  analysis,  all  other  factors  were  always 
statistically  significant.  That  is,  statistical  evidence 


Table  13. — Comparison  of  average  number  of  experts  identified  per  identi- 
fier, by  method,  occupation,  expertise,  and  experience 

Overall 
average 


Selection 
method 

number  of 
experts 

Occupation 
M/A  R/A 

Expertise 
Lower  Upper 

Years  of 
experience 

Total 

7.1 

nsd 

—  nsd  -- 

nss^ 

Top-one 

1.6 

nsd  -- 

--  nsd  -- 

nss 

Top-two 

2.9 

nsd 

--  nsd  -- 

nss 

Top-tfiree 

5.0 

nsd 

nsd  — 

nss 

Top-four 

5.3 

--  nsd  — 

nsd 

nss 

^Not  statistically  different  at  0.05  level. 
^Not  statistically  significant  at  0.05  level. 


Table  14.— One-factor  analysis  of  variance  showing  the  independent  influence  of  three  participant 
characteristics  on  elk  summer  cover  and  forage  quality  ratings 


Source  of  Degrees  of   Cover  Forage 

variation  freedom  SS  F  a  SS  F 


Experience  2  19,0  3.3  0.036  15.3  2.4  ns^ 

Expertise  1  128.9        45.5         <0.001  48.7  15.6  <0.001 

Occupation  1  61.1         21.3         <0.001  41.0  13.1  <0.001 

Error   varies  

Total  1,880  5,457.0  —  —  5,913.0  —  — 


'Statistically  nonsignificant  at  0.05  level. 
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exists  that  habitat  quality  ratings  varied  with  profes- 
sional characteristics  of  the  participating  expert.  Second, 
the  effects  of  these  factors  were  extremely  small.  Were  it 
not  for  our  desire  to  assess  the  implications  of  the  effect 
professional  status  has  on  ratings,  further  analyses  of 
professional  status  would  have  been  discontinued.  Instead, 
only  the  effect  of  the  experience  factor  was  judged  to  be 
so  small  that  it  would  not  be  considered  further. 

To  provide  a  direct  comparison  between  the  effect  of 
professional  status  and  vegetation  characteristics  on 
habitat  quality  ratings,  a  series  of  three-factor  analyses 
were  conducted.  Expertise  or  occupation  was  treated 
(separately)  as  one  factor,  and  combination  of  habitat 
type  groups  and  cover  types  (HT/CT),  along  with  combi- 
nation tree  size  and  stand  density  classes  (TS/SD),  con- 
stituted the  other  two  factors.  Analyses  of  variance  indi- 
cated that  the  occupation  or  expertise  factors  are 
statistically  significant.  However,  the  effect  was  small. 
Whereas  the  factors  pertaining  to  vegetation  type  and 
structure  accounted  for  70  to  80  percent  of  the  variation 
in  habitat  quality  ratings,  professional  status  accounted 
for  2  percent  or  less.  Nevertheless,  cover  and  forage  rat- 
ings do  vary  (at  a  statistically  significant  level),  depend- 
ing on  the  participant's  occupational  status  or  level  of 
expertise.  How  much  and  in  what  way  were  they  differ- 
ent? Based  on  subsequent  investigations,  the  hsting 
below  provides  some  indication  of  these  differences: 

Cover  Forage 

Researchers/academics  compai-ed  to 

managers/administrators                      -1-0.4  -1-0.3 
More  expertise  compared  to 

less  expertise                                      -I-.5  -I-.3 

On  the  average,  R/A  systematically  rate  elk  habitat 
higher  than  do  M/A.  Similarly,  the  more  expert  par- 
ticipants rate  habitat  higher  than  the  less  expert.  The 
magnitude  of  these  effects  should  be  interpreted  both  in 
the  context  of  the  seven-level  rating  scale  and  overall 
rating  means,  3.6  for  cover  and  4.1  for  forage  Whether 
differences  of  these  magnitudes  are  of  practical  sig- 
nificance is  judgment  that  lies  beyond  the  purview  of 
this  report.  This  much  can  be  said:  if  only  the  less 
expert  or  only  R/A  had  been  selected  as  participants, 
resulting  habitat  ratings  would  have  been  systematically 
different.  Caution:  because  there  is  a  high  correlation 
between  the  occupational  and  expertise  classes,  the 
differences  shown  above  are  not  additive. 

How  many  participants  are  needed  for  a  Delphi  experi- 
ment? That  depends  on  a  number  of  factors,  particularly 
the  cost  of  participation,  importance  of  the  results,  and 
the  inherent  variability  of  the  topic.  A  relatively  large 
number  of  experts  was  selected  to  participate  in  this 
study  so  that  the  effect  of  smaller  groups  could  be 
assessed. 

One  critical  assumption  must  be  made  in  comparing 
alternative  numbers  of  participants:  the  nature  of  the 
feedback  would  not  be  affected  by  the  number  of  par- 
ticipants. This  assumption  permitted  us  to  neutralize  the 
effect  of  feedback  and  simulate  results  if  fewer  than  11 
experts  had  participated.  To  do  this,  results  obtained 
after  the  third  iteration  with  11  participants  (the 
assumed  standard)  were  compared  to  results  that  would 


have  been  obtained  with  a  lesser  number  of  participants. 
The  third  iteration  median  involving  the  11  experts 
amounts  to  consensual  "truth"  within  the  context  of 
this  Delphi  application.  First  iteration  results  for  each 
expert  were  individually  compared  to  that  standard 
when  prospects  of  a  single  participant  were  evaluated. 

Figure  8  shows  a  comparison  between  the  third  itera- 
tion standard  and  the  range  of  results  that  could  have 
occurred  with  different  numbers  of  participants.  For 
example,  with  two  participants,  the  results  could  have 
ranged  from  an  average  deviation  of  more  than  1.0, 
about  75  percent  above  the  standard,  to  as  httle  as  0.2, 
nearly  50  percent  below  the  standard.  If  seven  or  more 
participants  were  used,  the  results  would  have  been 
within  25  percent  of  the  standard.  Not  all  subsets  of 
participants  necessarily  differ  from  the  total  11.  Indeed, 
there  are  specific  groups  of  two,  four,  six,  and  more  par- 
ticipants that  could  have  been  used  such  that  their 
median  ratings  would  have  been  virtually  identical  to 
those  of  the  11.  However,  the  problem  is  that  no  a  priori 
basis  exists  to  select  the  "correct"  individuals  for  any 
subset. 

These  comparisons  tend  to  mask  the  great  differences 
in  study  results  that  could  have  been  obtained  if  fewer 
participants  had  been  used.  To  illustrate,  consider  the 
extreme  case  of  one  participant.  Figure  8  shows  that  if 
only  one  participant  were  used,  that  person's  response 
could  have  been  as  much  as  130  percent  of  the  standard. 
Depending  on  which  of  the  1 1  individuals  was  asked  to 
assess  the  forage  quality  of  the  mountain  grassland  situ- 
ation, response  would  have  ranged  from  "very  low"  to 
"very  high"  quality,  the  maximum  possible  range.  While 
ranges  for  other  situations  were  not  necessarily  that 
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Figure  8.— Range  of  deviations  from  final 
consensus  for  elk  summer  cover  and  forage 
quality  ratings,  by  alternative  number  of 
participants. 
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large,  they  were  typically  large  enough  to  cast  doubt  on 
the  wisdom  of  relying  on  a  single  individual's 
assessment. 

The  study  team  had  expected  large  differences 
between  the  standard  and  small  number  of  participants, 
but  that  as  the  number  of  participants  increased,  ratings 
would  converge  to  the  third  iteration,  1 1 -participant 
standard.  Instead,  the  convergence  pattern  shown  in  fig- 
ure 8  is  nearly  linear.  Based  on  our  research,  we  con- 
clude that  more  participants  are  better  than  less,  at 
least  up  to  11. 

Effects  on  Consensus.— The  ultimate  objective  in  any 
Delphi  application  is  to  achieve  or  promote  consensus 
among  participants  regarding  some  topic.  While  the 
word  consensus  has  an  intuitive  meaning  (general  agree- 
ment), it  has  no  direct  quantitative  interpretation.  As 
discussed  earher,  the  quantitative  measure  of  consensus 
we  developed  for  each  elk  habitat  setting  (PSR)  was  the 
mean  deviation  from  the  median.  Iteration,  participant 
characteristics,  number  of  participants,  and  more,  will  be 
discussed,  all  regarding  their  effect  on  consensus. 

As  the  study  progressed  from  the  first  to  the  third 
iteration,  there  was  convergence  toward  consensus,  but 
overall  habitat  quality  ratings  did  not  change.  Table  15 
shows  that  the  overall  average  of  the  medians  did  not 
change  with  iteration.  The  same  is  also  true  for 
individual  PSR's.  With  each  iteration  there  were  171 
opportunities  for  median  ratings  to  change.  In  fact,  only 
seven  cover  medians  changed  between  the  first  and  sec- 
ond iterations  and  only  four  changed  between  the  second 
and  third.  Forage  medians  changed  even  less:  five 
between  the  first  and  second  and  two  between  the  sec- 
ond and  third.  In  no  case  did  any  median  change  by 
more  than  one  class.  Basically,  the  first  iteration 
provided  a  near-perfect  approximation  of  the  final 
results.  If  the  purpose  of  the  Delphi  exercise  were  sim- 
ply to  determine  median  habitat  quality  ratings,  one 
iteration  would  have  been  adequate. 

But  Delphi  projects  not  only  attempt  to  assess  some 
topic,  they  also  strive  to  promote  consensus  among  par- 
ticipants. Consensus  enhances  credibility.  The  second 
point  shown  in  table  15  is  the  consistent  pattern  of 
improvement  in  consensus  for  both  summer  cover  and 
forage.  The  overall  average  deviation  from  the  individual 
medians  after  the  first  iteration  was  about  0.8  of  a  class. 
This  reduced  by  about  a  third  between  the  first  and  sec- 
ond iterations,  a  statistically  significant  improvement  in 
consensus.  Average  deviation  from  the  medians  dropped 


again,  by  about  15  percent,  between  the  second  and 
third  iterations— once  more  a  statistically  significant 
change.  Changes  between  the  first  and  second  iterations 
accounted  for  over  three-fourths  of  the  total  convergence 
toward  consensus.  We  strongly  suspect  that  if  a  fourth 
iteration  took  place,  virtually  no  additional  improvement 
in  our  measure  of  consensus  would  have  occurred.  The 
fact  that  both  the  ultimate  level  of  consensus  and  con- 
sensus improvement  for  forage  were  better  than  cover 
surprised  us.  Communication  with  participants  specifi- 
cally indicated  that  they  were  having  much  more  diffi- 
culty with  forage  ratings  than  cover. 

The  pattern  of  group  movement  toward  consensus 
resulted  from  individual  response  changes  by  each  par- 
ticipant. The  range  of  differences  between  individual  par- 
ticipants and  the  group  norm  was  substantial.  Consider 
the  case  of  first  iteration  cover,  the  largest  range.  While 
the  overall  group  average  deviation  from  the  medians 
was  0.82,  individual  average  deviations  from  the  group 
norm  ranged  from  as  low  as  0.50  to  as  high  as  1.29, 
more  than  a  twofold  difference.  By  the  third  iteration 
the  range  had  decreased  considerably,  to  a  low  of  0.07 
and  a  high  of  1.05.  Similar  patterns  can  be  noted  for  for- 
age. The  lowest  individual  average  deviation  from  the 
medians  was  0.07,  for  third  iteration  cover.  That  per- 
son's ratings  and  the  group's  median  cover  ratings 
agreed  for  158  of  the  171  elk  habitat  settings,  and  none 
of  the  13  differences  were  ever  by  more  than  one  class. 

As  each  participant  was  given  the  opportunity  to 
adjust  ratings,  the  ratings  were  generally  adjusted 
toward  the  group  norm.  The  largest  individual  move- 
ment toward  group  consensus  occurred  between  the  first 
and  second  iterations  where  the  differences  between  one 
participant's  judgment  and  the  overall  medians  were 
reduced  by  64.7  percent,  almost  two-thirds.  But  more 
interestingly,  some  participants  diverged  from  the  over- 
all medians.  The  largest  divergence  occurred  between  the 
second  and  third  iterations  for  cover  where  the  differ- 
ence between  one  of  the  participants  and  the  group  norm 
increased  by  more  than  35  percent.  Therefore,  the  overall 
pattern  of  group  convergence  is  a  real  mixture  of 
individual  participants  converging  toward  and  diverging 
from  the  group  norm. 

We  conducted  several  statistical  analyses  on  the  effect 
of  participant  characteristics  on  consensus.  All  analyses 
treated  the  third  iteration  mean  deviation  from  the 
medians  as  the  dependent  variable.  Categories  for  the 
independent  factors  of  occupational  status,  expertise 


Table  15.— Average  median  and  mean  deviation  from  medians  for  elk 
summer  cover  and  forage  quality  ratings,  by  Delphi  iteration 


Characteristic 

Iteration 

Cover 

Forage 

Average  median 

1 

13.61-1 

4.07-1 

2 

3.62  1 

4.09  1 

3 

3.63J 

4.10-1 

Mean  deviation 

1 

.82 

.80 

from  medians 

2 

.60 

.50 

3 

.52 

.42 

Brackets  denote  no  statistical  difference  at  0.05  level. 
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level,  and  experience  were  as  before.  Two  additional 
independent  factors  were  added— confidence  and  time. 
One  of  the  questions  asked  in  the  exit  interview  ques- 
tionnaire dealt  with  how  confident  participants  were 
with  cover  and  forage  results.  Responses  were  handled 
as  categories  such  as  "very  confident,"  "confident,"  and 
so  forth.  The  study  team  also  had  records  on  how  much 
time  participants  devoted  to  their  assigned  Delphi  tasks. 
Time  was  handled  as  continuous  variables  in  simple  lin- 
ear regression  analyses. 

Analysis  results  shown  in  table  16  indicate  that  the 
level  of  consensus  achieved  for  each  participant  was  not 
related  to  the  participant's  occupation,  expertise, 
experience,  or  the  confidence  each  had  in  study  results. 
In  terms  of  consensus,  managers/administrators  did  not 
differ  from  researchers/academics  by  a  statistically  sig- 
nificant amount.  Participants  expressing  more  confi- 
dence in  study  results  did  not  differ  from  those  express- 
ing less.  These  results  tend  to  confirm  an  advantage 
ascribed  to  Delphi— it  overcomes  participant  status 
influence.  Status  can  be  based  on  occupation,  expertise, 
or  experience.  However,  movement  toward  consensus 
was  significantly  related  to  the  amount  of  time  devoted 
to  Delphi  tasks.  Highly  significant  regression  models 
explained  about  55  percent  of  this  variation  in  con- 
sensus, simply  on  the  basis  of  time.  Our  study  results 
indicated  that  movement  toward  consensus  was  related 
to  what  the  participants  did,  not  who  they  were. 

The  magnitude  of  measured  consensus  is  intimately 
tied  to  the  measurement  scale,  a  seven-level  scale  in  this 
study.  As  the  number  of  levels  on  the  measurement 
scale  increases,  so  do  the  opportunities  for  choice  or  dis- 
agreements. Hence,  one  would  expect  more  consensus 
with  fewer  scale  levels  than  with  more.  To  assess  the 
effect  of  measurement  scale  change  on  consensus,  an 
analytical  measure  was  needed  that  could  simultaneously 
reflect  both  changes  in  rating  variability  and  changes  in 
rating  scale.  The  coefficient  of  variation  (CV|.  expressed 
as  a  percentage,  is  such  a  measure: 

%CV  =  4  100 
X 

where  "s"  denotes  standard  deviation  and  "X"  the 
mean.  Whereas  mean  deviation  from  the  median  (MDM) 
is  an  absolute  measure  of  consensus,  %CV  is  a  relative 
measure,  allowing  comparisons  between  different  config- 
urations of  the  rating  scale. 


Table  16.— One-factor  analysis  of  variance  results  showing 
extent  to  which  consensus  (mean  deviation  from 
medians)  varies  with  five  independent  participant 
characteristics 


Independent 
participant 


characteristic 

Cover 

Forage 

Occupation 

No 

No 

Expertise 

No 

No 

Experience 

No 

No 

Confidence  in  results 

No 

No 

Time  devoted 

Yes 

Yes 

Different  rating  scale  configurations  amount  to  differ- 
ent arrangements  of  the  grouping  or  sorting  process. 
Because  the  participants  were  instructed  to  follow  a 
specific  procedure,  habitat  ratings  could  be  rearranged 
to  simulate  four  sorting  outcomes— the  original  seven- 
level  scale  (D),  two  five-level  scales  (B  and  Cl.  and  a 
three-level  scale  (A),  as  shown  in  figure  9. 

Habitat  ratings  were  recoded,  as  appropriate,  to  simu- 
late the  three  new  grouping  arrangements.  Original  and 
recoded  ratings  were  used  to  compute  'cCV  for  each  of 
the  171  PSR's.  A  two-factor  analysis  of  variance  was 
conducted  where  %CV  was  the  dependent  variable,  and 
the  four  grouping  arrangements  along  with  the  three 
iterations  constituted  levels  of  the  two  independent  fac- 
tors. The  data  sets  contained  2,052  observations  (171 
PSR's  X  3  iterations  X  4  grouping  arrangements).  Anal- 
ysis results  indicate  that  the  iteration  means  and  group- 
ing means  were  significantly  different  while  their  interac- 
tion was  not.  The  result  pertaining  to  iterations  was 
expected  because  statistically  significant  movement 
toward  consensus  with  increased  iterations  has  already 
been  demonstrated  and  discussed.  The  fact  that  statisti- 
cally significant  differences  were  found  between  group- 
ing means,  but  that  the  interaction  between  grouping 
and  iteration  was  not  significant,  signifies  that  the 
differences  in  group  means  did  not  depend  on  iteration. 
The  differences  were  present  in  all  iterations. 

Based  on  the  analysis  of  variance  just  discussed,  the 
listing  below  provides  insight  into  the  consequences  of 
alternative  grouping  arrangements: 


Grouping 
arrangement 
A 
B 
C 
D 


%cv 


Cover 

28 
r23 
28-" 
21 


Forage 

21  n 

16 
26 
19  J 


The  lower  the  "^"tCV,  the  smaller  the  relative  variation. 
Brackets  indicate  nonsignificant  differences.  Note  first 
that  not  all  "^fCV  for  various  groupings  are  significantly 
different  from  each  other.  Most  surprisingly,  the  forage 
%C'V  for  grouping  A  and  grouping  D  are  not  statisti- 
cally different.  That  means  that  relative  consensus 
would  not  have  been  increased  if  the  participants  sorted 
into  three-class  scale  consisting  of  low.  medium,  and 
high  habitat  quality  classes  rather  than  into  the  seven 
classes  that  were  actually  used.  However,  a  marked 
statistical  increase  should  be  noted  for  cover  rating.  The 
%CV  value  of  28  percent  was  reduced  to  about  21  per- 
cent with  movement  from  grouping  A  to  grouping  D. 
This  degree  of  increase  toward  consensus  associated 
with  different  grouping  arrangements  is  roughly  equiva- 
lent to  the  increase  observed  when  going  from  the  first 
to  the  third  iteration.  Substantial  increases  toward  con- 
sensus are  apparently  possible  by  the  use  of  a  scale  with 
fewer  classes,  but  the  possible  increases  may  not  actu- 
ally take  place.  We  know  of  no  way  to  distinguish 
beforehand  between  situations  where  increases  toward 
consensus  would  take  place  (as  for  cover)  and  situations 
where  consensus  either  is  not  increased  or  is  actually 
decreased  (as  for  forage,  especially  grouping  C). 
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Figure  9.— Four  alternative  grouping 
arrangements. 


Data-Gathering  Instrument  and  Feedback.— There 
were  several  unique  ways  by  which  the  Delphi  process 
was  implemented  in  this  study.  The  data-gathering 
instrument  was  one.  It  consisted  mainly  of  pictorial  site 
representations  (FSR's)  and  the  Q-sort  process.  Addition- 
ally, the  method  of  providing  feedback  was  somewhat 
unique.  This  section  presents  our  findings  on  these 
topics.  The  basis  for  our  findings  was  the  exit  interview 
questionnaire  administered  after  all  habitat  quality  rat- 
ings had  been  obtained. 

The  first  thing  participants  dealt  with  was  the  set  of 
171  PSR's  representing  a  variety  of  elk  habitat  settings. 
Each  PSR  consisted  of  three  parts— a  photocopy  of  a 
color  photograph  depicting  the  setting,  a  sketch  illus- 
trating a  cross-sectional  view  of  the  vegetation,  and  a 
written  narrative  description  of  the  setting.  We  asked 
the  participants  how  adequate  these  components  were  as 
a  basis  for  rendering  judgment  on  habitat  quality.  Table 
17  shows  the  percent  adequacy  associated  with  various 
combinations  of  the  three  components  with  respect  to 
cover  and  forage.  First,  the  vast  majority  of  participants 
felt  that  together  all  three  components  were  adequate  to 
accomplish  the  needed  tasks.  Second,  the  photocopy  was 
consistently  judged  to  be  inferior  to  either  the  sketch  or 
the  narrative.  While  the  combination  of  the  sketch  and 
narrative  was  highly  adequate  for  both  cover  and  forage 
ratings,  the  narrative  was  clearly  more  useful.  In  the 
case  of  forage,  the  narrative  alone  was  almost  as  ade- 
quate as  it  was  in  combination  with  any  other  compo- 
nent. On  the  other  hand,  the  narrative  and  the  sketch 
were  needed  to  provide  the  highest  indication  of  ade- 
quacy for  cover  ratings. 

These  results  are  consistent  with  previous  results. 
Earlier  analysis  of  variance  discussions  showed  that 
vegetation  structure  was  important  to  understanding 


Table  17.— Percentage  distribution  of  adequacy  judgments  for 
pictorial  site  representation  (PSR)  components  and 
component  combinations,  by  elk  summer  cover  and 
forage 


Component  and 
combination 

Percent  adequacy 

Cover 

Forage 

Single  component 

Photo  (P) 

27.3 

18.2 

Sketch  (S) 

54.6 

45.5 

Narrative  (N) 

63.6 

72.7 

Two  components 

P  and  S 

27.3 

18.2 

P  and  N 

36.4 

63.6 

S  and  N 

81.8 

63.6 

Three  components 

81.8 

81.8 

cover  ratings  and  that  the  type  of  vegetation  played  a 
dominant  role  in  forage  ratings.  The  type  of  vegetation 
was  primarily  described  in  the  PSR's  by  the  narrative, 
which  was  the  most  important  PSR  component  for  for- 
age ratings.  Information  on  vegetation  structure  was 
clearly  shown  in  the  sketch,  but  was  also  contained  in 
the  narrative.  The  combination  of  these  two  components 
was  judged  adequate  by  most  participants  for  cover 
ratings. 

The  participants'  low  regard  for  the  photo  component 
was  not  expected.  Was  the  negative  reaction  due  to  pho- 
tographic quahty?  Probably  not,  in  our  study.  We  asked 
the  participants  to  compare  several  alternative  ways  of 
constructing  the  photo  component  on  a  five-level  scale 
ranging  from  "much  inferior"  to  "equivalent"  to  "much 
superior."  Comparisons  were  made  between  color  photo- 
copy, color  print,  and  black  and  white  prints.  While 
about  half  the  respondents  thought  color  prints  would 
have  been  "superior"  to  color  photocopies,  the  other  half 
thought  they  would  be  "equivalent."  On  the  other  hand, 
over  90  percent  of  the  participants  indicated  that  either 
color  photocopies  or  color  prints  were  "superior"  or 
"much  superior"  to  black  and  white  prints.  The  lack  of 
support  for  the  photo  component  was  probably  unrelated 
to  topics  of  color  versus  black  and  white  or  print  versus 
photocopy.  Rather,  several  participants  complained  that 
the  subject  matter  portrayed  in  the  photocopy  did  not 
correspond  to  the  topic  of  the  PSR.  The  problem  was 
not  that  the  photocopy  was  inconsistent  with  the  PSR 
topic,  but  rather  that  the  photocopy  was  insufficient. 
Each  photocopy  represented  only  one  of  a  large  number 
of  possible  portrayals,  all  of  which  were  consistent  with 
the  topic.  For  example,  while  our  PSR  may  accurately 
depict  a  seedling/sapling  stand,  numerous  such  stands 
do  not  look  like  the  one  shown  in  the  PSR.  The  problem 
was  that  the  topic  was  far  broader  than  that  which 
could  be  captured  in  a  single  photo,  regardless  of 
whether  it  was  a  print  or  photocopy,  color,  or  black  and 
white. 

The  next  thing  participants  had  to  deal  with  was  the 
sorting  process,  the  Q-sort  technique.  When  the  study 
team  chose  to  use  a  sorting  procedure  that  resulted  in 
each  PSR  being  placed  into  one  of  seven  habitat  quality 
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groups,  the  concern  was  whether  seven  groups  were  too 
few  to  adequately  assess  habitat  quality.  The  counter- 
vailing pressure  was  that  it  is  more  difficult  to  sort 
items  into  more  groups.  We  asked  the  participants  to 
judge  both  the  ease  and  the  adequacy  for  alternative 
grouping  arrangements.  Because  the  patterns  of 
response  pertaining  to  cover  and  forage  were  virtually 
the  same,  they  were  combined.  Table  18  shows  a 
progression  from  "very  easy"  to  "very  difficult"  and 
from  "too  general"  to  "far  too  specific"  as  the  number 
of  sorting  groups  increased  from  three  to  seven.  The 
median  participant  assessment  for  the  three-group 
arrangement  was  "very  easy"  and  "too  general."  The 
median  assessment  for  the  seven  groups  actually  used 
was  "difficult"  and  "too  specific."  We  strongly  suspect 
that  if  more  than  seven  groups  were  used,  the  median 
assessment  would  have  been  "very  difficult"  and  "far 
too  specific." 

The  final  Delphi-related  topic  was  feedback.  Recall 
there  is  no  standard  format  for  providing  feedback  in 
the  Delphi  process  and  that  the  study  team  chose  to 
provide  feedback  in  the  form  of  summary  evaluations 
and  rating  printouts.  The  exit  interview  questionnaire 
asked  participants  how  useful  the  feedback  was  in  help- 
ing them  reevaluate  PSR's  and  whether  the  amount  of 
feedback  was  correct.  Table  19  shows  an  interesting  pat- 
tern of  response,  again  combining  the  near-identical 
response  pertaining  to  cover  and  forage.  Participants 
were  almost  evenly  divided  on  the  amount  of  feedback 
provided;  about  half  said  it  was  "about  right"  and  about 
half  said  it  was  "too  much."  This  applied  both  to  the 
summary  evaluations  and  the  rating  printouts.  Useful- 
ness was  a  different  story.  The  summary  evaluations 
were  generally  judged  to  be  either  "neutral"  or  "useful." 
The  rating  printouts  were  not  as  well  received.  While 
almost  one  in  five  found  the  printouts  "very  useful," 
about  a  third  of  the  participants  found  the  printouts 
"harmful."  W^e  interpret  the  response  of  "harmful"  as  a 
participant's  reaction  against  the  pressure  to  change 
earlier  ratings  and  thus  conform  to  the  group  norm. 

Time  Costs.— Costs  must  be  assessed  when  evaluating 
Delphi  or  any  other  information-developing  procedure. 
There  are  two  kinds  of  costs— those  borne  by  the  study 
team  and  those  borne  by  the  participants.  Study  team 
costs  consisted  of  expenses  and  personnel  time.  But 
because  so  many  aspects  of  this  study  were  driven  by 
research  considerations,  rather  than  simply  data  gather- 
ing, detailed  cost  records  were  not  maintained.  Principal 
expense  items,  however,  include  salaries,  costs  to  pro- 
duce the  sets  of  PSR,  mailing,  and  computer  use  for 
data  analysis. 

Participant  costs  consisted  entirely  of  their  time. 
Based  on  their  detailed  records,  participants  on  the  aver- 
age, each  devoted  about  13  hours  and  20  minutes  to  this 
study.  Table  20  shows  that  the  second  phase  (forage) 
took  about  a  fourth  less  time  than  the  first  phase 
(cover),  and  that  each  respective  iteration  generally  took 
less  time.  There  was  a  substantial  range  in  the  amount 
of  time  individual  participants  devoted  to  the  study— as 
much  as  nearly  a  tenfold  difference  in  the  case  of  third- 
iteration  cover.  We  point  to  an  earlier  result;  the  degree 
of  consensus  achieved  was  strongly  related  to  the 


Table  18.— Percentage  distribution  of  judgments  of  ease  in 
sorting  and  adequacy  in  classifying  for  four 
alternative  grouping  arrangements,  cover  and  forage 
phases  combined 

 Grouping  arrangements  

Distribution  D:3  C:5  B:5  A:7 

of  judgments  strata       strata       strata  strata 


 Percent 

Ease 


Very  easy 

54.6 

Easy 

31.8 

36.4 

13.6 

Moderately  difficult 

9,1 

50.0 

59.1 

40.9 

Difficult 

4.6 

13.6 

27.3 

31.8 

Very  difficult 

27.3 

\dequacy 

Far  too  general 

36.4 

Too  general 

55.5 

59.1 

27.3 

About  right 

9.1 

36.4 

36.4 

31.8 

Too  specific 

4.5 

31.8 

40.9 

Far  too  specific 

4.5 

27.3 

Table  19.— Percentage  distribution  of  judgments  of  amount  and 
usefulness  of  feedback  information,  cover  and  for- 
age phases  combined 

 Type  of  feedback  

Distribution  Summary  Rating 

of  judgments  evaluations  printout 


 Percent 

Amount 


Far  too  little 

Too  little 

9.1 

About  right 

40.9 

45.4 

Too  much 

50.0 

54.6 

Far  too  much 

Usefulness 

Very  harmful 

Harmful 

9.1 

31,8 

Neutral 

40.9 

13.6 

Useful' 

50.0 

36.4 

Very  useful 

18.2 

Table  20. — Mean  and  minimum-maximum  for  participant  time 

involved  in  study,  by  phase 

Ptiase 

Mean 

MIn-max 

-  Minutes  

Cover  —1 

186.1 

79-345 

—2 

■  144.8 

60-337 

—3 

100.1 

20-195 

Total 

431.0 

190-862 

Forage  —1 

129.2 

70-260 

—  2 

129.6 

35-302 

—3 

77.8 

30-168 

Total 

336.6 

135-690 

Questionnaire 

32.2 

17-45 

Overall 

799.8 

345-1,546 
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amount  of  time  devoted  to  tasks;  overall,  those  par- 
ticipants tending  toward  the  low  end  of  the  time  range 
were  significantly  further  away  from  the  consensual 
standard  than  were  those  devoting  more  time. 

Information  provided  by  participants  allowed  a 
detailed  breakdown  of  the  time  devoted  to  specific 
activities  for  each  iteration.  The  listing  below  shows  an 
approximate  average  time  distribution  (cover  and  forage 
combined)  for  the  2.5  hours  devoted  to  first-iteration 
activities: 

Specific  activities  First  iteration 

Minutes 

Three-strata  sorting  56 
Seven-strata  sorting  39 
^  Sorting  review  30 

Evaluation  form  33 

158 

The  initial  three-strata  sorting  took  the  most  time- 
almost  an  hour— to  accomplish.  While  sorting,  par- 
ticipants kept  track  of  time  needed  to  sort  the  first  25 
PSR's,  the  second,  and  so  forth.  Respondents  averaged 
about  24  seconds  per  PSR  over  the  first  25;  this  dropped 
by  about  a  fourth  to  17  seconds  per  PSR  for  the  final 
sortings. 

The  listing  below  shows  the  approximate  distribution 
of  time  devoted  to  second  and  third  iteration  activities, 
cover  and  forage  combined: 

Second  and 

Specific  activities  third  iterations 

Minutes 

PSR  refamiliarization  10 
Study  rating  printout  8 
Study  summary  evaluations  8 
Reevaluate,  resort  62 
Evaluation  form  (second  iteration  only)  25 

1 13 

Several  weeks  usually  elapsed  between  iterations,  so  the 
10  minutes  devoted  to  the  first  activity  was  little  other 
than  getting  reacquainted  time.  Only  about  16  minutes 
on  the  average  were  devoted  to  reading  and  evaluating 
feedback  materials,  substantially  less  time  than  the 
study  team  devoted  to  preparing  materials.  Then  about 
an  hour  was  spent  resorting  or  rearranging  the  PSR's, 
presumably  on  the  basis  of  the  feedback  materials. 

DISCUSSION 

Our  study  objective  was  to  apply  and  evaluate  the 
Delphi  Method  as  a  means  for  acquiring  useful  informa- 
tion for  forest  management  decisions.  We  chose  to  focus 
on  elk  habitat  quality,  a  topic  that  was  both  meaningful 
and  complex.  One  advantage  of  the  Delphi  approach  is 
that  it  permits  investigations  into  a  broad  scope  of  sub- 
ject matter.  A  field  measurement  study  with  a  scope  as 
broad  as  this  Delphi-based  study  would  likely  prove 
impractical,  if  not  infeasible. 

It  is  difficult  to  judge  how  well  study  objectives  were 
met.  If  satisfaction  with  outcome  is  any  indication,  our 
purposes  were  accomphshed.  The  study  team  is  quite 
satisfied  with  the  study  results,  both  in  terms  of  increas- 


ing our  understanding  of  Delphi  and  our  understanding 
of  elk  habitat  quality.  The  participants  apparently 
shared  this  feeling.  One  of  the  questions  asked  in  the 
exit  interview  questionnaire  was  how  satisfied  par- 
ticipants were  with  the  Delphi  process.  Over  80  percent 
indicated  they  were  either  "satisfied"  or  "very  satis- 
fied," with  the  remainder  indicating  "neutral."  This 
degree  of  satisfaction  is  heartening  for  two  reasons. 
First,  we  doubt  that  a  well-designed  and  executed  field 
study  would  be  rated  substantially  different  from  this 
Delphi  application.  Second,  considering  that  all  par- 
ticipants had  extensive  background  in  biological  science 
and  its  inherent  field  orientation,  the  satisfaction  indi- 
cated for  a  social  science  methodology  is  somewhat 
unexpected. 

If  the  success  of  an  information-gathering  procedure 
can  be  judged  on  the  basis  of  confidence  in  results,  this 
Delphi  application  can  again  be  regarded  as  successful. 
The  exit  interview  questionnaire  also  asked  participants 
how  much  confidence  they  had  in  the  (anticipated)  final 
habitat  ratings,  given  knowledge  of  their  involvement 
and  the  project's  process  only.  Table  21  shows  that  the 
vast  majority  of  participants  were  "confident"  in  the 
habitat  quality  ratings,  both  for  cover  and  forage.  While 
the  participants  had  not  seen  the  final  outcomes,  they 
were  thoroughly  familiar  with  the  sorting  process,  feed- 
back materials,  and  reevaluations.  In  a  similar  manner, 
an  assessment  of  a  field-oriented  study  is  usually  judged 
on  the  basis  of  its  experimental  design,  measurement 
technique,  and  so  forth,  not  on  the  basis  of  the  actual 
measurements  obtained.  lii  that  regard,  we  suspect  that 
a  well-designed  and  executed  field  study  would  not  typi- 
cally rate  any  higher  than  this  Delphi  application. 

In  total,  this  study  obtained  11,286  measurements  and 
remeasurements  on  elk  habitat  quality.  Participants 
devoted  about  147  hours  to  providing  those  measure- 
ments, or  approximately  47  seconds  per  measurement 
and  remeasurement.  This  translates  to  slightly  over  18 
days  of  participant  time.  One  advantage  of  the  Delphi 
process  is  it  does  not  require  face-to-face  group  partici- 
pation. A  procedure  requiring  that  kind  of  interaction 
could  easily  consume  all  18  days  in  travel  time  alone.  If 
the  average  daily  wage  rate  for  participants  was  $200, 
this  study  would  have  cost  slightly  over  $3,600.  That 
amount  would  not  support  much  field-based  data  collec- 
tion. On  balance,  Delphi  appears  to  be  a  highly  cost- 
effective  means  of  obtaining  these  types  of  resource 
management  data. 


Table  21.— Percentage  distribution  of  judgment  of  confidence  in 
final,  anticipated,  study  results 


Confidence  Cover  Forage 


Very  confident  9.1  — 

Confident  72.7  81.8 

Neutral  —  18.2 

Nonconfident  18.2  — 

Very  nonconfident  —  — 

Total  100.0  100.0 
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Opportunities  to  use  Delphi  results  clearly  depend  on 
the  topic  addressed.  In  general,  data  gathered  by  the 
Delphi  process  could  be  used  like  comparable  data 
gathered  by  a  field  study  or  any  other  procedure.  For 
example,  the  habitat  quality  ratings  readily  lend  them- 
selves to  a  computer  mapping  routine  that  could  develop 
an  elk  habitat  quality  map  similar  to  a  soil  type 
map.  We  envision  plastic  overlays— one  for  cover  and 
another  for  forage.  These  displays  could  be  used  to 
assess  spatial  forage-cover  relationships  of  elk  habitat. 
Study  results  could  also  be  used  to  evaluate  manage- 
ment opportunities,  particularly  those  involving  vegeta- 
tion manipulation.  Would  replacing  a  sawtimber  stand 
with  seedling/saplings  improve  elk  forage?  If  only  a 
fixed  number  of  acres  could  be  thinned,  in  which  habitat 
types  should  the  thinning  be  concentrated  to  promote 
maximum  forage? 

Opportunities  to  extend  Delphi  to  other  applications 
seem  nearly  endless.  Opportunities  are  mostly  hmited  by 
the  knowledge,  skills,  and  imagination  of  the  persons 
applying  Delphi,  not  by  Delphi  itself.  We  can  readily 
envision  applying  the  process  to  questions  of  long-range 
environmental  consequences  of  timber  harvest  activities, 
habitat  requirements  for  grizzly  bear,  extramarket  valua- 
tion, and  more.  But  the  most  fruitful  opportunities  to 
extend  Delphi  to  other  applications  are  best  known  by 
natural  resource  managers  who  know  their  specific  infor- 
mation needs  and  priorities.  This  method  takes  time  and 
thoughtful  planning,  but  is  totally  implementable  by 
field  managers.  Results  of  this  study  should  increase 
awareness  of  Delphi  and  its  potential  applications. 

Earlier  in  this  paper  we  criticized  some  of  the  nonfield 
methods  being  used  to  develop  decision-making  informa- 
tion. We  feel  that  the  Delphi  process  is  far  more 
satisfactory— capable  of  giving  rise  to  more  credible, 
defensible  information.  Whether  or  not  it  is  the  best  non- 
field  data  collection  technique  will  probably  vary  with 
situation  and  circumstance.  But  Delphi  is  not  an  alterna- 
tive to  field  data  collection  and  scientific  experimenta- 
tion; quality  information  and  advances  in  scientific 
knowledge  and  understanding  can  result  only  from  scien- 
tific exploration  involving  onsite  measurement  and 
experimentation.  Yet  the  Delphi  process  need  not  be  in 
conflict  with  scientific  exploration.  The  two  can  be  com- 
plementary. Analysis  and  evaluation  of  Delphi-based 
information  can  provide  testable  hypotheses  for  subse- 
quent scientific  inquiry.  This  study's  results  contain 
many  such  hypotheses.  Moreover,  Delphi  could  be  used 
to  integrate  and  synthesize  the  current  state  of  scientific 
knowledge  to  answer  questions  and  address  issues  about 
which  the  state-of-the-art  science,  by  itself,  is  silent.  This 
may  well  be  an  important  use  of  the  Delphi  Method.  For 
as  nature  abhors  a  vacuum,  decision  makers  abhor 
silence.  When  decision-making  information  needs  cannot 
be  met  with  onsite  measurement  and  experimentation, 
Delphi  could  be  used  to  fill  the  void. 
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APPENDIX  A:  COMBINATIONS  OF  FOREST  HABITAT  TYPES  COMPRISING  HABITAT 
TYPE  GROUPS 


Habitat  types 

Ponderosa  pine/bluebunch  wheatgrass 
Ponderosa  pine/Idaho  fescue 

Douglas-fir/bluebunch  wheatgrass 
Ponderosa  pine/Idaho  fescue 
Douglas-fir/ninebark 

Douglas-fir/snowberry  (snowberry  and  pinegrass  phases) 

Douglas-fir/pinegrass  (pinegrass  and  kinnikinnick  phases) 
Douglas-fir/dwarf  huckleberry 

Douglas-fir/blue  huckleberry 
Douglas-fir/twinflower  (blue  huckleberry  phase) 

Douglas-fir/snowberry  (bluebunch  wheatgrass  phase) 
Douglas-fir/pinegrass  (bluebunch  wheatgrass  and 
ponderosa  pine  phases) 
Subalpine  f\rlClintonia 
Subalpine  fir/GaZ/um 
Subalpine  fir/twinfiower 
Subalpine  V\rlMenziesia 
Subalpine  fir/beargrass 
Subalpine  fir/wood  rush 
Subalpine  fir/grouse  whortleberry 
Spruce/horsetail 
SprucelClintonia 
Spruce/dwarf  huckleberry 

Grand  f'irlClintonia 
Western  hemlock/C//nfon/a 
Western  redcedar/(various  types) 

Whitebark  pine-subalpine  fir 
Whitebark  pine 

All  hardwoods 


Habitat  type  group 

Ponderosa  pine 

Douglas-fir  I 
Douglas-fir  II 
Douglas-fir  III 
Douglas-fir  IV 
Douglas-fir  V 

Subalpine  fir  I 

Subalpine  fir  II 
Subalpine  fir  III 
Subalpine  fir  IV 

Spruce 

Redcedar-grand  fir 

Whitebark  pine 

Bottonnlands 
Mountain  nneadow 
Mountain  grassland 
Mountain  brush 
Scree 
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APPENDIX  B:  MEAN  DEVIATIONS  FROM  MEDIANS  FOR  FINAL  ELK  SUMMER 

COVER  AND  FORAGE  QUALITY  RATINGS  BY  VEGETATION  COVER 
TYPE,  FOREST  HABITAT  TYPE  GROUPS,  STAND  DENSITY  CLASS, 
AND  TREE  SIZE  CLASS 


Tree  size  class 

Stand 

Cover 

ForsQG 

Vegetation 

Forest  habitat 

density 

Seedling/ 

Pole- 

Saw- 

Seedling/ 

Pole- 

Saw- 

cover  types 

type  groups 

class 

sapling 

timber 

timber 

sapling 

timber 

timber 

Hardwoods 

Bottomlands 

Poor 

0.5 

n  Q 

u.o 

0.7 

Medium 

0.6 

0.5 

Well 

.3 

.7 

.7 

.5 

Spruce 

Poor 

.4 

7 

o 
.o 

.4 

Medium 

.3 

.2 

Well 

.5 

.6 

.4 

.2 

Ponderosa  pine 

Ponderosa  pine 

Poor 

.3 

-1  n 

.1 

.3 

Medium 

.6 

.1 

Weil 

.5 

.9 

.3 

.2 

Douglas-fir  1 

Poor 

.5 

Q 

.y 

.2 

Medium 

.5 

.2 

Well 

.4 

.7 

.2 

.3 

Douglas-fir  II 

Poor 

.5 

.O 

.5 

Medium 

.5 

.4 

Well 

.5 

.6 

.5 

.5 

Douglas-fir  III 

Poor 

.3 

.O 

c 
.o 

.4 

Medium 

.5 

.7 

Well 

.4 

.5 

.5 

.5 

Douglas-fir  V 

Poor 

.5 

i  -1 
1 .  1 

.2 

Medium 

.5 

.4 

Well 

.4 

.6 

.2 

.3 

Scree 

Poor 

.1 

c 

.Q 

n 
u 

0 

Medium 

.5 

0 

Well 

.5 

.5 

0 

0 

Douglas-fir 

Douglas-fir  1 

Poor 

.4 

-D 

o 

.4 

Medium 

.3 

.2 

Well 

.4 

.6 

.5 

.4 

Douglas-fir  II 

Poor 

0 

-1  i 
1 .  1 

-O 

.3 

Medium 

.3 

.5 

Well 

.5 

.6 

.5 

.5 

Douglas-fir  III 

Poor 

.4 

1  .  1 

.0 

.4 

Medium 

.^ 

.5 

Well 

.5 

.5 

.4 

.4 

Douglas-fir  iV 

Poor 

.2 

c 

.0 

.5 

Medium 

.4 

.5 

Well 

.7 

.4 

.5 

.4 

Douglas-fir  V 

Poor 

.5 

.O 

.3 

- 

Medium 

.5 

.2 

Weil 

.4 

.5 

.1 

.2 

Subalpine  fir  1 

Poor 

.4 

C 

-D 

c 
.0 

.1 

Medium 

.4 

.6 

Well 

.3 

.5 

.4 

.1 

Scree 

Poor 

.1 

1.1 

0 

0 

Medium 

.8 

0 

Well 

.4 

.6 

0 

0 

Lodgepole  pine 

Douglas-fir  ill 

Poor 

.3 

.6 

.7 

.4 

Medium 

.3 

.5 

Well 

.3 

.3 

.5 

.5 

Douglas-fir  IV 

Poor 

.2 

.8 

.5 

.5 

Medium 

.4 

.5 

Well 

.3 

.4 

.3 

.5 

(con.) 
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APPENDIX  B.  (Con.) 


Tree  size  class 


Stand 

Cover 

Forage 

Vegetation 

Forest  naDitat 

density 

beeciling/ 

Pole- 

Saw- 

beediing/ 

Pole- 

Saw- 

cover  types 

type  groups 

class 

sapling 

timber 

timber 

sapling 

timber 

timber 

Subalpine  fir  i 

Poor 

.4 

.6 

.3 

.5 

Mpd  i  1 1  m 

1  V  1  C  U  1  LJ  1  1  1 

.3 

5 

Well 

.5 

.5 

.2 

.1 

opr  uoc 

Poor 

.0 

Q 

c 
.0 

A 

ivi  cu  1  urn 

Well 

5 

R 
.  o 

0 

.  ^ 

^iihdlriino  fir  II 
OUUciipiMcr  Ml  II 

Poor 

R 

.D 

.0 

KA  ^  H  i  1 1  m 
IVI  cU  1  U  1 1 1 

7 

.4 

Well 

.9 

4 

5 

5 

^iih^lnino  fir  III 
OU  Udl  p  1 1  Ic  III  III 

Poor 

c 

1  n 
1  .u 

c 

.0 

R 

IVI  c(J  1  U  III 

/I 

.4 

.0 

Well 

5 

4 

5 

R 

■  O 

Qiihalninfi  fir  l\/ 
OUUdipiiic  III  iv 

Pnn  r 

,\J 

q 

Q 
-O 

Q 
■  O 

KA  QrH  i  1  1  m 
IVI  cU  1  U 1 1 1 

c 

.o 

7 

Well 

.5 

.5 

.7 

7 

vVCoLCIII  IdlOII 

^iihalnino  fir  1 
OUUctipillt?  Ill  1 

Poor 
r  UUI 

1^ 

.o 

Q 
-O 

t\A  oH  i  1 1  m 
IVI  fcJU  1  U  M  1 

c 
.3 

c 
.0 

Well 

4 

5 

2 

4 

^IILjOlllldliM  opiu^c 

^iihalnino  fir  1 
OUUdipiiiu  III  1 

Poo  r 

KA  oH  i  1 1  m 
tVlcU  1  U  III 

o 
.o 

o 
.o 

Well 

.4 

4 

4 

-3 

.  vj 

Qiihalnino  fir  II 

OUUdipillC    III  II 

Poo  r 

R 

c 
.o 

7 

KA  oH  i  1 1  m 
IVI  cU  1  u  n  1 

c 
.o 

Well 

5 

1 

7 

CiiH\olr^inQ   ftr  l\/ 
O  U  Ud  1  p  1 1  1  C    III  IV 

Poor 

.D 

p 

.D 

.D 

KA  orl  i  1 1  m 
IVI  t;U  1  U  i  1  1 

■  O 

.0 

Well 

.3 

5 

6 

C  pi  r  1  1  ^ 

op  r  uoc 

Poor 

c 

.D 

Q 

.y 

.o 

c 
.0 

\A  oH  i  1 1  m 
IVI  cU  1  U  1  1  1 

7 

o 

.d. 

Well 

.4 

3 

2 

(^ranH  fir 

\A/pc;tprn  re^rirf^ci^rl 
vvcoidii  icjocudi/ 

Poor 

7 

.o 

.0 

n  ra  n  H  fir 
y  1  di  1  u  III 

KA  oH  i  1 1  m 

IVI  t7U  1  U  1  1  1 

7 

.0 

Well 

.8 

2 

4 

Wpc+prn  rpdrpfiar 

VVCOlCIM  ICUUCUdl/ 

Poor 

r  UUI 

q 
.  y 

c 
.0 

n  ra  n  H  fir 
y  I  d  1 1  u  III 

hA       i  1 1  m 
IVI  cU  1  U 1 1 1 

/I 

.H 

>i 

.4 

Well 

3 

7 

Si ihpi  1  ni np  fir 

\J  fj  kJ  CA\  IJ  I  I  1^  III 

^lih^ilninp  fir  II 

OUL-'dl^lllCr    III  1) 

Poor 

1  UUI 

R 

R 

.4 

^^  oH  i  1 1  m 
IVI  CU  1  U  1  1  1 

.D 

c 
.0 

Well 

.6 

2 

2 

Siih^^lninp  fir  III 

OULJCll|JlliC    III  ill 

Poor 

r  UU 1 

4 

R 

7 

R 

oH  i  1 1  m 
IVI  cU  1  U  1 11 

c 
.O 

.0 

Well 

,5 

6 

7 

4 

^iihainino  fir  l\/ 
ouudipiiio  III  Iv 

Poor 

r^uui 

c 
.o 

7 

7 

R 

.0 

^^  orl  1 1 1  m 
IVI  t7U  I  U 1 1 1 

A 

.0 

Well 

7 

7 

R 

\A/h itpharU"  nino 
vviiiicuairs.  jjiiic 

Ciil-iQlriino  fir  III 
OU  Udl  p  1 1  Icr  III  III 

Poor 

c 
.O 

1.1 

c 
.0 

.0 

Mp(i  i  1 1  m 

1  V  1      u  1  U  1  1  1 

Ci. 
.u 

Well 

.5 

.4 

.5 

.6 

vvniTeuarK  piriG 

Poor 

o 
.0 

.5 

.5 

Medium 

.5 

.5 

Well 

.5 

.5 

.5 

.5 

Forbs  and  grasses 

Mountain  meadow 

.1 

0 

Mountain  grassland 

.1 

.9 

Shrubs 

Mountain  brusti 

.6 

.5 
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APPENDIX  C:  THREE-CLASS  DICHOTOMOUS  KEY  FOR  DISCRIMINANT  ANALYSIS 
OF  ELK  SUMMER  COVER  AND  FORAGE  QUALITY 

Cover 

If:  +4.20  +2.79(UD1)  +2.20(TS1)  +0.73(M1)  -10.60(C2)  -9.79(C3)  -7.31(C4)  -11.00(C5)  >  0 -►Low 

otherwise 


If:  +3.45  +1.97(TS1)  -1.39(TS2)  +3.01(M1)  -1.12(T1)  -1.88(C3)  -7.02(C4)  >  O^Medium 

otherwise 

I 

High 

.  .  .  Overall  correct  classification:  84% 


Forage 

If:  -14.17  +14.58(UD1)  +2.77(E1)  +9.27(TS1)  -2.50(TS2)  -4.35(M1)  +1.08(C2)  >  0-^Low 

K 

otherwise 


If:  -17.97  -1.45(UT1)  +20.91(TS1)  +17.74(TS2)  +4.37(M1)  +18.01(T1)  +6.46(C1)  +8.33(C2)  +5.78(C3) 

+  4.82(C4)  +7.48(C5)  >  0-^ Medium 

\ 

otherwise 

\ 

High 

.  .  .  Overall  correct  classification:  92% 
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The  Delphi  Method:  application  to  elk  habitat  quality.  Research  Paper  INT-353. 
Ogden,  UT:  U.S.  Department  of  Agriculture,  Forest  Service,  Intermountain 
Research  Station;  1985.  32  p. 

Study  was  designed  not  only  to  apply  and  evaluate  the  Delphi  Method  but 
also  to  nneasure  elk  habitat  quality.  Elk  habitat  was  restricted  to  summer  cover 
and  forage  in  western  Montana.  Eleven  elk  habitat  experts  participated  in  rating 
habitat  quality.  Statistical  analyses  were  conducted,  relating  habitat  quality  to 
factors  such  as  cover  type  and  stand  density.  Analyses  of  the  Delphi  Method 
pertained  to  consensus,  number  of  participants,  and  more. 


KEYWORDS:  Delphi,  information,  decision-making,  elk  habitat,  wildlife 
management 


INTERMOUNTAIN  RESEARCH  STATION 


The  Intermountain  Research  Station  provides  scientific  l<nowledge  and 
technology  to  improve  management,  protection,  and  use  of  the  forests  and 
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